【Deep Learning OCR Series·6】In-depth analysis of CRNN architecture

## Introduction CRNN (Convolutional Recurrent Neural Network) ni imwe mu nyubako z'ingenzi mu bijyanye no kwiga byimbitse OCR, byatanzwe na Bai Xiang et al. mu 2015. CRNN ihuza ubuhanga bwo gukuramo ibice bya convolutional neural networks (CNNs) hamwe nubushobozi bwo gukurikirana imiyoboro ya neural isubiramo (RNNs) kugirango igere kumenyekanisha inyandiko ya nyuma. This article will provide an in-depth analysis of CRNN's architecture design, working principles, training methods, and specific applications in OCR, providing readers with a comprehensive technical understanding. ## Overview of CRNN Architecture ### Design Motivation Mbere ya CRNN, sisitemu ya OCR ubusanzwe yakoreshaga uburyo butandukanye: kugenzura imiterere no gutandukanya byabanje gukorwa, hanyuma buri nyuguti imenyekana. Ubu buryo bugira ibibazo bikurikira: *Limitations of Traditional Methods**: - Error propagation: Errors in character segmentation can directly affect recognition results - Complexity: Requires designing complex character segmentation algorithms - Poor robustness: Sensitive to character spacing and font changes - Kunanirwa guhangana n'imiterere ihoraho: Ikibazo cy'imiterere ihoraho mu nyandiko yanditswe n'intoki biragoye gutandukanya **CRNN's Innovative Ideas**: - End-to-end learning: Mapping directly from images to text sequences - No Segmentation: Avoids the complexity of character segmentation - Sequence Modeling: Use RNNs to model dependencies between characters - CTC Alignment: Addresses input-output sequence length mismatches ### Ubwubatsi rusange Urwego rw'Igihugu rw'Ubugenzacyaha (RIB) rugizwe n'ibice bitatu by'ingenzi: **1. Convolutional Layers**: - Function: Extract feature sequences from input images - Input: Text line image (fixed height, variable width) - Output: Feature map sequence **2. Recurrent Layers**: - Function: Model contextual dependencies in feature sequences - Input: The feature sequence extracted by the CNN - Output: A feature sequence with contextual information **3. Transcription Layer**: - Function: Convert feature sequences to text sequences - Method: Using CTC (Connectionist Temporal Classification) - Output: The final text recognition result ## Sobanukirwa n'ibisobanuro birambuye by'ibinyabiziga by'ibinyabiziga ### Feature Extraction Strategies Igishushanyo mbonera cy'Umujyi wa Kigali cyashyizweho by'umwihariko mu rwego rwo kumenyekanisha ibitabo by'imyandikire y'ikiremwamuntu: **Network Structure Features**: - Shallow Depth: 7 layers of convolutional layers are usually used - Small convolutional kernels: 3×3 convolutional kernels are mainly used - Pooling Strategy: Use pooling sparingly in the width direction **Specific Network Configuration**: Input: 32×W×1 (Height 32, Width W, Single Channel) Conv1: 64 3×3 convolutional nuclei, step 1, fill 1 MaxPool1: 2×2 pools, step length 2 Conv2: 128 3×3 convolutional kernels, step 1, fill 1 MaxPool2: 2×2 pooled, step size 2 Conv3: 256 3×3 convolutional nuclei, step 1, fill 1 Conv4: 256 3×3 convolutional cores, step 1, fill 1 MaxPool3: 2×1 pooled, step size (2,1) Conv5: 512 3×3 convolutional cores, step 1, fill 1 BatchNorm + ReLU Conv6: 512 3×3 convolutional kernels, step 1, fill 1 BatchNorm + ReLU MaxPool4: 2×1 pooled, step size (2,1) Conv7: 512 2×2 convolutional nuclei, step 1, fill 0 Output: 512×1×W/4 ### Key Design Considerations **High Compression Strategy**: - Intego: Compress the image to 1 pixel high - Method: Compress buhoro buhoro uburebure ukoresheje ibice byinshi bya pooling - Impamvu: Uburebure bw'umurongo w'inyandiko nta gaciro buhagije **Width Holding Strategy**: - Intego: Kubungabunga amakuru y'ubugari bw'ifoto uko bishoboka kose - Method: Reduce pooling operations in the width direction - Impamvu: Amakuru y'uruhererekane rw'inyandiko agaragara ahanini mu cyerekezo cy'ubugari **Feature Map Conversion**: Ibiciro by'ibicuruzwa - Raw Output: C×H×W (Channel × Height× Width) - Converted: W×C (Sequence Length× Feature Dimension) - Method: Take the feature vector for each width position as a time step ## Sobanukirwa n'ibisobanuro byimbitse by'inyandiko y'umuhanzi Kizito Mihigo ### RNN Selection CRNNs usually use bidirectional LSTMs as the loop layer: *Bye Bye Bidirectional LSTM**: - Contextual Information: Use both forward and backward context - Kwishingikiriza kure: LSTM ifite ubushobozi bwo guhangana n'imiterere ya kure - Gradient Stabilization: Avoids the problem of gradient disappearance **Network Configuration**: Input: W×512 (sequence length × feature dimension) BiLSTM1: ingirabuzimafatizo 256 zihishe (128 imbere + 128 inyuma) BiLSTM2: ingirabuzimafatizo 256 zihishe (128 imbere + 128 inyuma) Output: W×256 (sequence length× hidden dimensions) ### Sequence Modeling Mechanisms **Timing Dependency Modeling**: Urwego rw'Igihugu rw'Ubugenzacyaha (RIB) rugaragaza imiterere y'amakimbirane hagati y'amakipe y'ibihugu byombi - Amakuru y'inyuguti y'ubushize afasha mu kumenya imiterere y'imiterere y'ubu - Information for following characters can also provide useful context - The information of the whole word or phrase helps to disambiguate **Feature Enhancements**: Ibicuruzwa byashyizweho na RNC bifite imiterere ikurikira: - Context-sensitive: Every location's features contain contextual information - Timing consistency: Features in adjacent locations have a certain continuity - Semantic richness: Combines visual and sequence features ## Ibisobanuro birambuye by'ibisobanuro by'inyandiko y'inyandiko ### CTC mechanism CTC (Connectionist Temporal Classification) ni igice cy'ingenzi cya CRNN: **Uruhare rwa CTCs**: - Addressing Alignment Issues: Input sequence lengths do not match output sequence lengths - End-to-end training: No need for character-level alignment annotations - Handle duplicates: Handle cases of duplicate characters correct **Uko CTC ikora**: 1. Expand the label set: Add blank labels on top of the original character set 2. Path Enumeration: Enumerates all possible alignment paths 3. Path Probability: Calculate the probability of each path 4. Marginalization: sum the probabilities of all paths to obtain the sequence probability ### CTC loss function **Mathematical Representation**: Ukurikije uruhererekane rw'ibicuruzwa X n'uruhererekane rw'intego Y, igihombo cya CTC gisobanurwa nka: L_CTC = -log P(Y| X) where P(Y| (b) Mu byiciro byose by'ubudehe, hashobora gushyirwaho uburyo bwose bushoboka bwo kugenzura imikoreshereze y'ibicuruzwa: P(Y| X) = Σ_π∈B^(-1)(Y) P(π| X) Here B^(-1)(Y) represents all the sets of paths that can be mapped to the target sequence Y. **Forward-Backward Algorithm**: Kugirango ubashe kubara neza igihombo cya CTC, algorithm yimbere-inyuma ya porogaramu ya dynamic ikoreshwa : - Forward Algorithm: Calculates the probability of reaching each state - Backward Algorithm: Calculates the probability from each state to the end - Gradient Calculation: Calculate gradients in combination with forward-backward probability ## CRNN Training Strategy ### Data preprocessing **Image Preprocessing**: - Size normalization: Unify the image height to 32 pixels - Aspect Ratio Maintenance: Keeps the aspect ratio of the original image - Grayscale Conversion: Convert to a single-channel grayscale image - Numerical normalization: Agaciro ka pixel gasanzwe kuri [0,1] cyangwa [-1,1] **Data Enhancement**: - Geometric transformations: rotation, tilt, perspective transformation - Lighting changes: brightness, contrast adjustment - Noise addition: Gaussian noise, salt and pepper noise - Blur: Motion blur, Gaussian blur ### Training Techniques **Learning Rate Schedule**: - Initial Learning Rate: Usually set to 0.001 - Decay Strategy: Exponential decay or step decay - Warm-up strategy:The first few epochs use a small learning rate **Regularization Techniques**: - Dropout: Add a dropout after the RNN layer - Weight degradation: L2 regularization prevent overfitting - Batch normalization: Use batch normalization in the CNN layer **Optimizer Selection**: - Adam: Adaptive learning rate, fast convergence - RMSprop: Suitable for RNN training - SGD+Momentum: Traditional but stable option ## Optimization and improvement of CRNN ### Architecture optimization **CNN Partial Improvements**: - ResNet Connections: Added residual connections to improve training stability - DenseNet Fabric: Dense connections improve feature multiplexing - Attention Mechanism: Introduces spatial attention in CNNs **RNN Partial Improvements**: - GRU Replacement: Use GRU to reduce the amount of parameters - Transformer: Replaces RNNs using self-attention mechanisms - Multi-Scale Features: Include features from different scales ### Performance Optimization **Inference Acceleration**: - Model Quantization: INT8 quantization reduces computational effort - Model pruning: Remove unimportant connections - Knowledge Distillation: Learn the knowledge of large models with small models **Memory Optimization**: - Gradient checkpoints: Reduce memory footprint during training - Mixed Precision: Train with FP16 - Dynamic graph optimization: Optimize the structure of the calculated graph ## Real-World Application Cases ### Handwritten text recognition **Application Scenarios**: - Digitize handwritten notes - Form autofill - Historical document recognition **Technical Features**: - Large character variation: Requires strong feature extraction capabilities - Continuous Stroke Processing: The benefits of the CTC mechanism are obvious - Context Matters: RNNs' sequence modeling capabilities are critical ### Printed text recognition **Application Scenarios**: - Digitize documents - Indangamuntu y'amatike - Ibimenyetso byo kumenyekana **Technical Features**: - Font Regularity: CNN feature extraction is relatively straightforward - Typography rules: Layout information can be used - High Accuracy Requirements: Requires fine model tuning ### Scene text recognition **Application Scenarios**: - Street View Text Recognition - Product label identification - Traffic sign recognition **Technical Features**: - Complex Background: Requires robust feature extraction - Severe deformation: Robust architecture design is required - Real-Time Requirements: Requires efficient reasoning ## Summary As a classic architecture of deep learning OCR, CRNN successfully solves many problems of traditional OCR methods. Uburyo bwayo bw'amahugurwa bwa nyuma, igishushanyo mbonera kidafite imiterere y'imiterere, no gutangiza uburyo bwa CTC byose bitanga icyitegererezo cy'ingenzi mu iterambere rya tekinoroji ya OCR. **Key Contributions**: - Kwigira kw'iherezo: Koroshya igishushanyo mbonera cya sisitemu ya OCR - Sequence Modeling: Effectively utilizes the sequence properties of text - CTC Alignment: Addressed sequence length mismatch - Igishushanyo cyoroshye: Byoroshye gusobanukirwa no gushyira mu bikorwa **Icyerekezo cy'iterambere**: - Attention Mechanism: Introducing attention to improve performance - Transformer: Replaces RNNs with self-attention - Multimodal fusion: Guhuza andi makuru nk'imiterere y'ururimi - Lightweight design: model compression for mobile devices Intsinzi ya CRNN ni igihamya cy'ubushobozi buhambaye bwo kwiga byimbitse mu rwego rwa OCR kandi itanga ubunararibonye bw'ingirakamaro bwo gusobanukirwa uburyo bwo gutegura uburyo bwiza bwo kwiga bwa nyuma. Mu gice gikurikiyeho tuzabagezaho ibijyanye n'imiterere n'imiterere y'imiterere y'umubiri w'umuntu.