A distributed PDF text recognition system built on AWS, processing pages in parallel across Lambda functions to eliminate multi-day batch bottlenecks.
NodeJS, Serverless framework, AWS Step Functions, AWS Lambda, tesseract-ocr, Poppler, Webpack
VirtualPostMail, a US-based virtual mailbox and registered agent service operating since 2009
S3-OCR was the third project TMS Outsource delivered for VirtualPostMail, following the Angular-to-React migration of the customer-facing UI and the VPM Alice back-office desktop application.
PDF-OCR is an application that performs OCR processing of a given PDF file in the fastest way possible. VirtualPostMail was using an older application for this purpose, but that application was not able to process large PDFs, it was not compatible with all the servers available, and frequently it needed 2 to 3 days to process a request, or sometimes was not able to finish it at all.
Our partners from VirtualPostMail relied on us to build a new application with improved performance to assure their customers have a fast and pleasant experience with their services. The new system had to handle large files reliably, process them quickly, and integrate automatically with the rest of the VirtualPostMail platform.
Our job was to build a new application from scratch. The new app is designed to work on the Amazon Web Services (AWS) platform using the Serverless framework. Large PDFs are now split between different servers and pages are processed in parallel, resulting in significantly lower processing time, while the costs remain the same.
We leveraged AWS cloud solutions to build a sophisticated application with increased flexibility, scalability, and reliability. S3-OCR is integrated with other VirtualPostMail systems so that the entire process is done automatically: a PDF arrives, the pipeline splits it, processes each page in parallel, and returns the recognized text without manual intervention.
AWS Step Functions coordinating Lambda into a parallel serverless workflow.
After the agreement with the client about the purpose and requirements of the application, we started by making an architecture diagram of the app. This helps system designers and developers visualize the high-level, overall structure to ensure that the system meets our goals before any code is written.
We decided to use AWS Step Functions, which lets you coordinate multiple AWS services into serverless workflows so you can build and update apps quickly. Using Step Functions, we made our workflows run fast using AWS Lambda for this feature-rich application.
Properly set up Step Functions automatically trigger and track each step, and retry when there are errors, so the application executes in order and as expected. This removes the need for manual monitoring and intervention during processing.
Large PDFs are split into individual pages, each processed by a separate Lambda function running tesseract-ocr and Poppler. Pages run in parallel rather than sequentially, which is what turns days of processing into minutes.
The before and after, in concrete terms.
Old system processing time for large PDFs, sometimes failing to finish at all.
Parallel Lambda execution processes pages simultaneously, with costs remaining the same.
Pages processed simultaneously across Lambda functions instead of sequentially on a single server
Performance improved dramatically while infrastructure costs remained the same
Step Functions automatically retry failed steps, so the pipeline completes without manual intervention
S3-OCR is one of three projects TMS Outsource built for VirtualPostMail.
Tell us about your product idea or your team's capacity gap. We'll outline the right approach for your situation.
Request a Free ConsultationCrypto market data analysis platform aggregating multi-exchange data with React and PHP.
