Novel approaches to verify the correctness of data extraction from scanned documents

OVERVIEW

ExxonMobil, the Seeker for this Wazoku Crowd Challenge, is looking for new approaches to verify the accuracy of the extracted data from scanned complex technical documents.

Reliable geoscience data is crucial for decision-making in many areas of economic and social activities, such as infrastructure development.

Given the sheer volume of the geoscience data—measured literally in millions of input pieces—the process of the data collection has been almost completely digitized. However, the Machine Learning-powered Optical Character Recognition (OCR) technology for text and structure extraction that lies in the core of this process is prone to errors.

Unfortunately, the process of quality control of the text and structure extraction is complicated by the lack of objective metrics allowing to estimate its accuracy, which results in the need of the follow-up manual control, which is resource-extensive and error-prone.

The goal of this Challenge, therefore, is to identify objective, measurable criteria allowing to estimate, in an automatic manner, the accuracy of the text and structure extraction from scanned documents—and indicate, when appropriate, the need for human verification and correction.

The Solvers will be provided with a set of images that can be used to develop and test their solutions.

By taking part in this Prize Challenge, you are granting ExxonMobil a right to use your submitted information; however ExxonMobil must determine award winners within 60 days from the start of evaluation otherwise they lose this right of use.

The Challenge requires a written proposal to be submitted. There will be a guaranteed award pool of $25,000, with at least one award of $15,000 or larger and no award being smaller than $2,500. Award distribution (or allocation) will be contingent upon the theoretical evaluation of the proposals. 

In this Challenge, Solvers may:

• Submit ideas of their own
• Submit third-party information that they have the right to use and further, the authority to grant ExxonMobil this right with the right to use and develop derivative works
• Submit information considered in the public domain without any limitations on use.

Submissions to this Challenge must be received by 11:59 PM (US Eastern Time) on March 25th, 2025.

Please review the later Participation Guidance section before submitting a proposal.

- Login and register your interest to start solving!

ABOUT THE SEEKER & ELIGIBILITY
 

ExxonMobil manages an industry-leading portfolio of resources and is one of the largest integrated fuels, lubricants and chemical companies in the world. We are focused on meeting society’s evolving needs for energy and essential products and reducing greenhouse gas emissions. We are focused on developing meaningful growth opportunities for our employees, and working to help our industrial, commercial, and retail customers succeed by leading in operational and financial performance, building essential partnerships, leveraging our advantaged portfolio, and creating innovative solutions.

Employees of ­­­­ExxonMobil are ineligible to receive an award for this Challenge.

THE CHALLENGE

Background

Collecting and analyzing reliable geoscience data is crucial for decision-making in many areas of economic and social activities:

• Subsurface resource exploration
• Drilling planning
• Infrastructure planning and development

Insufficient and erroneous data can erode business value, which places a high premium on the quality of the data and the accuracy of its collection.

Unfortunately, given the sheer volume of the geoscience data—measured literally in millions of input pieces—the process of the data collection has become complicated. Another problem arises from the occasional low quality of the scanned source documents.

Long gone are days when the geoscience data was processed manually; now, this process is almost completely digitized.

A typical data collection flow consists of the following steps:

• Paper reports and well logs, the two most common formats in which geoscience data is recorded, are hand-scanned.
• The text and structure of the documents are processed by document analysis tools, for example, the Optical Character Recognition (OCR) software (Azure document intelligence and AWS Textract being other popular systems) and converted into unstructured digital text.
• Unstructured digital text is then converted into standardized searchable formats.
• The data is incorporated into appropriate datasets and databases.

Crucial to the integrity of the whole process is the quality of the text and structure extraction.

Although the accuracy of modern OCR technology is quite high—with many commercial systems claiming accuracy rates of 99% and higher—it’s worth remembering that these accuracy numbers are often based on ideal conditions, such as clear, high-resolution scans of documents with standard fonts and clear formatting. Also, there can be complex formatting and hand-written text included.

In real-world scenarios, however, factors like document quality, font variations, complex layouts, and background noise can impact OCR accuracy. As a result, the actual accuracy rate may vary depending on the specific document and the OCR system being used—and is usually significantly lower than 99%.

Exacerbating the problem is the lack of robust approaches to quality control (QC) of the text and structure extraction. In particular, we lack objective metrics allowing us to estimate the OCR accuracy of scanned documents, both new ones and, even more importantly, older documents processed with earlier, less accurate, versions of the OCR software.

The Challenge

The goal of this Challenge, therefore, is to identify objective, measurable criteria allowing us to estimate the correctness of the OCR text and table extraction from scanned documents—and indicate, when appropriate, the need for human verification and correction.

We envision that a set of characters or signs will be identified signaling that the OCR-processed documents may contain errors (and, ideally, roughly estimate their rate). Possible examples of characters or signs may include:

1. Visual Clues (scanned source documents):

a. Unusual, unrecognizable, and missing (or extra) characters.
b. Misspelled or completely unrecognizable words.
c. Irregular formatting of the document (variability of font, size, spacing, etc.).

2. Content-Related Clues:

a. Presumably missing parts of the document.
b. Incoherent (difficult to understand or making no sense) text or numeric data.

3. Metadata from document analysis results:

a. Confidence scores on various levels.

Importantly, while we’re interested in any specific character of sign signaling inaccuracy of the OCR text extraction, we’re looking for the solution that will propose an automatic QC process (that is, with only minimal, if at all, human intervention). Ideally, code allowing to execute such a QC check digitally will accompany the proposed solution. Such solutions will receive the greatest attention.

To assist the Solvers in developing their solutions, we’re providing some real-life data that can serve as a training/testing data set.

The dataset comprises 54 technical documents regarding exploration activities in Costa Rica. These paper documents containing tables, paragraphs, figures/charts, and handwritten texts/equations/tables have been manually scanned and stored in PDF format, (see sample pages in Fig. 1).

The OCR results, extracted using the Azure Document Intelligence Layout model (https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/prebuilt/layout?view=doc-intel-4.0.0&tabs=sample-code), are saved in JSON format. Text, tables, document structures with various confidence scores are available in the results.

Feel free to leverage any information in the OCR results in your proposed approach. The source PDF file and OCR JSON file can be paired using the first number in their filenames.

Please note that the readability of the text can fluctuate within a single document and even on a single page. The aggregated score of the entire document/page is not considered sufficient for QC purpose.

You can access the files via the ShareFile link (you will be asked to create an account and password to be able to download the data): https://exxonmobil.sharefile.com/i/ib539806b91146b18  

Figure 1. Sample pages of the Training/Testing data set.

SOLUTION REQUIREMENTS

We’re open to any innovative idea, however unorthodox, for as long as the proposed solutions meets the following Solution Requirements:

1. The estimate of the correctness of the OCR text extraction will require only the scanned document itself and not the original document.
2. Ideally, the proposed solution would be able to make a granular estimate of the quality of various text groupings within a document.
3. The proposed solution should also provide quality metrics on table information that has been extracted using advanced OCR model.
4. The solution should not require an unreasonable amount of compute power to execute
5. The proposed solution should use commonly available hardware devices (scanning and computer) and not require expensive, custom-built equipment.
6. Ideally, the proposed solution will be fully automatic, that is, require only minimal, if at all, human intervention. In this case, a code allowing to execute such a solution should be presented.

Solutions with any Technology Readiness Level (TRL) are invited, although those with TRL 4-6 are preferred. Solutions with higher TRLs will receive more support.

This Prize Challenge has the following features:

1. By taking part, you are granting ExxonMobil a right to use your submitted information; however ExxonMobil must determine the award winners within 60 days from the start of evaluation otherwise they lose this right of use. You will receive notification.

2. There will be a guaranteed award pool of $25,000, with at least one award of $15,000 or larger and no award being smaller than $2,500. 

3. The award distribution will be determined after theoretical evaluation of the proposals by ExxonMobil.

4. Solvers may:

• Submit ideas of their own,
• Submit third-party information that they have the right to use and further, the authority to convey to ExxonMobil this right with the right to use and develop derivative works,
• Submit information considered in the public domain without any limitations on use.

5. ExxonMobil may also issue “Honourable Mention” recognitions for notable submissions that are not selected for monetary awards.

ExxonMobil may wish to collaborate with the Solver at the conclusion of the Challenge. Please indicate your interest in collaborating.
 

YOUR SUBMISSION

Please login and register your interest, to complete the submission form.

The submitted proposals must be written in English, and in your submission form response and attachments, you should include:

1. Participation type – you will first be asked to inform us how you are participating in this challenge, as a Solver (Individual) or Solver (Organization).
2. Solution Level - the Technology Readiness Level (TRL) of your solution.
3. Collaborating- there may be an opportunity to collaborate at the conclusion of this Challenge. Please indicate if collaborating is of interest to you.
4. Problem & Opportunity - highlight the innovation in your approach to the Problem, its point of difference, and the specific advantages/benefits this brings (up to 500 words).
5. Solution Overview - detail the features of your solution and how they address the Solution Requirements (500 words, there is space to add more in the summary field below, and to add any appropriate supporting data, diagrams, etc.). Include code if available.
6. Experience - Expertise, use cases and skills you or your organization have in relation to your proposed solution (up to 500 words).
7. Solution Risks - any risks you see with your solution and how you would plan for this (up to 500 words).
8. Timeline, capability and costs - describe what you think is required to deliver the solution, estimated time and cost (up to 500 words).
9. Online References - provide links to any publications, articles or press releases of relevance (up to 500 words).

PARTICIPATION GUIDANCE

1. Submission Close Date: Submissions to this Challenge must be received by 11:59 PM (US Eastern Time) on March 25th, 2025.
2. Late submissions: Late submissions will not be considered.
3. Submission form and attachments: Your submission will be evaluated by the evaluation team first reviewing the information and content you have submitted at the submission form, with attachments used as additional context to your form submission. Submissions relying solely on attachments will receive less attention from the evaluation team.
4. Evaluation notification steps: After the Challenge submission close date, ExxonMobil will complete the review process and make a decision with regards to the winning solution(s) according to the timeline in the Challenge header. All Solvers who submit a proposal will be notified about the status of their submissions.
5. Use of AI: Please note that any submissions produced solely with generative AI are not of interest.
6. Learn more: Find out more about participation in Wazoku Crowd Challenges.