Novel Approaches to Improving the Quality of Knowledge Graphs

OVERVIEW

The Seeker for this Wazoku Crowd Challenge is looking for new approaches to improving the quality of Knowledge Graphs (KGs) as a tool for organizing complex information.

Knowledge Graphs (KGs), which are structured representation of knowledge that organizes information about real-life entities (objects, places, concepts, etc.) and their interrelations, are growing in popularity as a powerful modern data and knowledge management tool.

KGs are particularly useful for integrating diverse datasets and providing context to information, making it more easily accessible to both human users and machine learning applications.

Central to the process of KG building is Ontology Development, which provides the foundational framework that defines the concepts, relationships, and rules within a specific domain. This structured representation enables effective data integration, sharing, and reasoning across various applications.

As with every complex process, Ontology Development is prone to mistakes that can lead to low quality KG building, lowering information retrieval quality.

Unfortunately, current quality control tools to assess the effectiveness of the Ontology Development process require manual approaches and are therefore difficult to scale. There is an urgent need to develop new approaches to assess the quality of ontology in a simple and intuitive way.

Developing these novel approaches is the objective of this Challenge, and the Solvers will be provided with examples of KGs that the Solvers can use to develop and test their solutions.

By taking part in and submitting to this Prize Challenge, you are granting the Seeker a right to use your submitted information, provided that the Seeker decides the award winners within 45 days from the start of evaluation, otherwise the Seeker retains the right to use only Awarded solutions.

This Prize Challenge requires a written proposal to be submitted. There will be a guaranteed award pool of $25,000, with at least one award of $10,000 or larger and no award being smaller than $2,500. Award distribution (or allocation) will be contingent upon the theoretical evaluation of the proposals. 

Solvers may:

• Submit ideas of their own
• Submit third-party information that they have the right to use and further, the authority to convey to the Seeker this right with the right to use and develop derivative works
• Submit information considered in the public domain without any limitations on use

Submissions to this Challenge must be received by 11:59 PM (US Eastern Time) on May 12th, 2025.

Please review the later Participation Guidance section before submitting a proposal.

- Login and register your interest to start solving!

Find out more about participation in Wazoku Crowd Challenges.

THE CHALLENGE

Background

Knowledge Graphs (KGs) are growing in popularity as a powerful modern tool for organizing complex information.

A KG is a structured representation of knowledge, in a graphic format, that organizes information about real-life entities (people, places, concepts, etc.) and their interrelations. In KGs, entities are connected to other entities through lines of structured data and special markup (often called schema markup). By capturing relationships and hierarchies among entities, knowledge graphs provide deeper insights compared to traditional data representations. One can characterize KGs as a combination of data with its meaning.

This is what makes KGs particularly useful for integrating diverse datasets and providing context to information, making it more easily accessible to both human users and machine learning applications (of which search engines and recommendation algorithms are the most prominent).

Building a typical KG involves the following major steps:

1. Data Collection: Gathering data from various sources, including databases, documents, and APIs.
2. Entity Recognition: Identifying and extracting entities from the collected data using various techniques, such as natural language processing (NLP).
3. Relationship Mapping: Determining how these entities are related to one another through semantic enrichment processes—by defining the relationships as “many-to-one” or “one-to-many”—which improve the meanings of the entities and relationships.
4. Ontology Development: Creating a structured framework that defines the types of entities and relationships within the graph.
5. Graph Database Implementation: Storing the knowledge graph in a graph database optimized for handling interconnected data.

Crucial to the whole process of KG building is Ontology Development.

Ontology is a formal and systematic representation of knowledge that specifies the types of entities, their properties, and the relationships between them within a particular domain. Ontology development is therefore a crucial aspect of constructing knowledge graphs, as it provides the foundational framework that defines the concepts, relationships, and rules within a specific domain. This structured representation enables effective data integration, sharing, and reasoning across various applications.

It's this aspect of ontology that makes KGs such a flexible, reusable—and easily acceptable to new data, definitions, and requirements—data layer used for answering complex queries across data silos.

Obviously, the key role ontology plays in the development of high-quality KGs poses strict requirements to the quality of the ontology development process itself. However, as a complex process, Ontology Development is prone to mistakes, of which the two most common are ambiguous definitions (failing to clearly define classes and properties, which can lead to confusion and misinterpretation) and inconsistent relationships (defining relationships that contradict each other or do not follow logical rules).  

These mistakes can lead to several negative outcomes, such as user confusion, ineffective reasoning, lower adaptability to new requirements and data sources, and reduced interoperability (the decreased ability of different systems to share and integrate data effectively).

The Challenge

Unfortunately, currently existing tools for the automated generation and quality control (QC) of ontologies—such as engaging domain experts to review the ontology or using ontology validation tools (e.g., Protégé)—are difficult to scale. There is an urgent need therefore to develop new approaches to automatically build and assess the quality of ontologies in a simple and intuitive way. This is the objective of this Challenge.

The most common way to evaluate ontologies is to use a set of competency questions. These questions include common facts and information that we would expect an artificial intelligence system to be able to correctly answer using a defined ontology and a knowledge graph. The quality of the ontology can then be measured based on the number of validation questions that it answers correctly. The answers to these questions will also give indication as to which subject matters the ontology is not able to handle effectively.

While we’re interested in any specific sign of potential KG mistakes, we’re looking for a solution that will propose ontologies, and evaluate them with minimal human intervention. We would ideally like this solution to propose an ontology, test this ontology on a set of validation questions, update the ontology definition, and propose a new ontology on this basis. An example of the process is shown below.  

We would welcome a technical description of how you would solve this problem. However, if you are more of a hands-on Solver, we would appreciate a solution including source code (or pseudocode) that can explicitly demonstrate the veracity of your approach.

Here are examples of two ontologies that include competency questions and source documents so that they can be re-built using your solution.

• Gene Ontology (GO) - https://www.geneontology.org/
• Plant Ontology (PO) - https://archive.plantontology.org/

These two ontologies are open source and contain documentation on how they were constructed. They also include a list of competency questions to determine how well the ontology works.

SOLUTION REQUIREMENTS

We’re open to any innovative idea, however novel and unorthodox, for as long as the proposed solutions meet the following Solution Requirements:

1. The estimate of the accuracy of KGs will require only KGs themselves and not the manual inspection of the Ontology Development process.
2. The proposed solution would be able to point to the aspects of the Ontology Development process in need of improvement.
3. The solution should not require an unreasonable amount of computing power to execute.
4. The proposed solution should use commonly available hardware devices (scanning and computers) and not require expensive, custom-built equipment.
5. Ideally, the proposed solution will be fully automatic, that is, require only minimal, if at all, human intervention. In this case, a code allowing to execute such a solution should be presented.

At the same time, we’re not going to accept solutions that:

1. Are based entirely on commercially available products.
2. Summarize commercial optionality only.

Solutions with any Technology Readiness Level (TRL) are invited, although those with TRL 2-6 are preferred.

This Prize Challenge has the following features:

1. By taking part and submitting, you are granting the Seeker a right to use your submitted information, provided that the Seeker decides the award winners within 45 days from the start of evaluation, otherwise the Seeker retains the right to use only Awarded solutions.You will receive notification.

2. There will be a guaranteed award pool of $25,000, with at least one award of $10,000 or larger and no award being smaller than $2,500. 

3. The award distribution will be determined after theoretical evaluation of the proposals by the Seeker.

4. Solvers may:

• Submit ideas of their own,
• Submit third-party information that they have the right to use and further, the authority to convey to the Seeker this right with the right to use and develop derivative works,
• Submit information considered in the public domain without any limitations on use.

5. The Seeker may also issue “Honorable Mention” recognitions for notable submissions that are not selected for monetary awards.

6. The Seeker may wish to partner with the Solver at the conclusion of the Challenge. Please indicate your interest in partnering.
 

YOUR SUBMISSION

Please login and register your interest, to complete the submission form.

The submitted proposals must be written in English, and in your submission form response and attachments, you should include:

1. Participation type – you will first be asked to inform us how you are participating in this challenge, as a Solver (Individual) or Solver (Organization).
2. Solution Level - the Technology Readiness Level (TRL) of your solution.
3. Partnering - there may be an opportunity to partner at the conclusion of this Challenge. Please indicate if partnering is of interest to you.
4. Problem & Opportunity - highlight the innovation in your approach to the Problem, its point of difference, and the specific advantages/benefits this brings (up to 500 words).
5. Solution Overview - detail the features of your solution and how they address the Solution Requirements (500 words, there is space to add more in the summary field below, and to add any appropriate supporting data, diagrams, etc.). Include code if available.
6. Experience - Expertise, use cases and skills you or your organization have in relation to your proposed solution (up to 500 words).
7. Solution Risks - any risks you see with your solution and how you would plan for this (up to 500 words).
8. Timeline, capability and costs - describe what you think is required to deliver the solution, estimated time and cost (up to 500 words).
9. Online References - provide links to any publications, articles or press releases of relevance (up to 500 words).

PARTICIPATION GUIDANCE

1. Submission Close Date: Submissions to this Challenge must be received by 11:59 PM (US Eastern Time) on May 12th, 2025.
2. Late submissions: Late submissions will not be considered.
3. Multiple submissions, 3 Maximum: In case of multiple submissions by the same Solver, only 3 submissions – the final 3 submitted – will be considered. Any other submissions will be deleted prior to evaluation.
4. Submission form and attachments: Your submission will be evaluated by the evaluation team first reviewing the information and content you have submitted at the submission form, with attachments used as additional context to your form submission. Submissions relying solely on attachments will receive less attention from the evaluation team.
5. Evaluation notification steps: After the Challenge submission close date the Seeker will complete the review process and make a decision with regards to the winning solution(s) according to the timeline in the Challenge header. All Solvers who submit a proposal will be notified about the status of their submissions.
6. Use of AI: Please note that any submissions produced solely with generative AI are not of interest.
7. Learn more: Find out more about participation in Wazoku Crowd Challenges.