Knowledge Graphs

Exercise 4.6 - Creating a Knowledge by Integrating Data

Exercise 4.6. Create a knowledge graph of companies by linking the records across Wikidata and the SEC data considered as part the problem 3.7. To connect the two data sets, you can either implement your own record linker, or use any available open-source tools. Report on the quality of the resulting data set in terms of the precision and recall of identifying the correct links across the SEC company data and the Wikidata company information.

Extracting Companies from Wikidata

To scope the project, we will begin by extracting information about the companies in the S&P 100 index. A curated list of these companies is available in Wikipedia. We start by simply scraping this list, and incorporating the list of tickers in our code.

To locate a company in Wikidata, we can search either by the company name or by its ticker symbol. A search based on the ticker symbol is generally more accurate because ticker symbols uniquely identify companies within a stock exchange. We will begin by explaining the statement model in Wikidata that is necessary to correctly query the ticker symbols. We next explain the approach for building a mapping table that relates a ticker symbol to each company. Finally, we show how to extract the properties of companies from Wikidata.

Understaing Wikdata Statements

In Wikidata, the association between a company and its ticker symbol is represented as a statement (also called a claim) in addition to a simple RDF triple. This allows additional information, such as the stock exchange on which the ticker symbol is used, the date on which the listing became effective, and supporting references, to be attached to the statement. Understanding this representation is helpful when writing SPARQL queries to extract information from Wikidata. Detailed documentation is available as part of official Wikidata RDF dump specification, but we present below a simplified discussion relevant to the current task.

Each Wikidata statement consists of a main value (the primary fact being asserted) together with optional qualifiers, references, and a rank. The main value captures the core assertion; qualifiers provide contextual details such as time, location, or applicability; references record the sources supporting the assertion; and the rank indicates whether the statement is preferred, normal, or deprecated. This statement model allows Wikidata to represent multiple, potentially conflicting, assertions about the same property while preserving their provenance and context.

As an example, consider the property ticker symbol (P249) for Apple. One statement records the ticker symbol "AAPL". A company may have different ticker symbols on different stock exchanges, so the ticker symbol alone is not sufficient. A qualifier such as stock exchange (P414) specifies that the ticker symbol "AAPL" is used on the NASDAQ exchange. The statement may also include references, such as an official NASDAQ listing or an SEC filing, documenting the source of the information. Together, the ticker symbol, its qualifiers, and its references form a complete statement describing where the ticker symbol is valid and how the information is supported.

Because a statement consists of several components, Wikidata defines multiple property namespaces that allow SPARQL queries to access each component independently. The prefix wd: denotes Wikidata entities, such as companies, people, and stock exchanges. The remaining namespaces all use the same property identifier but interpret it differently depending on the part of the statement being accessed. For example, the property identifier P249 always refers to the ticker symbol property. When prefixed with wdt:, it retrieves the ticker symbol directly; when prefixed with p:, it links a company to the corresponding statement; when prefixed with ps:, it retrieves the statement's main value; when prefixed with pq:, it retrieves qualifiers associated with the statement; and when prefixed with pr:, it retrieves properties of the statement's references. Additional variants, such as psv:, pqv:, and prv:, return structured value objects for complex datatypes. Thus, the namespace determines how the property is interpreted, while the property identifier determines what property is being accessed.

The different property namespaces can be illustrated using Apple's ticker symbol (P249). A simple query uses the direct property wdt:P249, for example, wd:Q312 wdt:P249 "AAPL", which directly returns Apple's ticker symbol. To retrieve additional information, the query first follows the claim property to a statement node using wd:Q312 p:P249 ?statement. The main value is then obtained using ?statement ps:P249 "AAPL". Qualifiers are attached to the same statement node; for example, ?statement pq:P414 wd:Q82059 indicates that the ticker symbol applies to the NASDAQ stock exchange. References are linked from the statement node using ?statement prov:wasDerivedFrom ?reference, where properties such as ?reference pr:P248 wd:Q... identify the source supporting the statement. Thus, each namespace corresponds to a different stage in traversing the statement: wdt: retrieves the fact directly, p: reaches the statement, ps: retrieves the statement's main value, pq: accesses its qualifiers, and pr: describes its supporting references.

Mapping Ticker Symbols to Wikidata Identifiers

It took several iterations to formulate the query in the sample code that starts from a ticker symbol and obtains its Wikidata identifier. The biggest problem is that the many ticker symbols have an associated end date, that is, they are no longer current, and must be excluded from our results. The query did not return a match for the ticker symbol BNY, because it has been recently updated, and on the date this code was run, Wikidata had not been updated. To correct for this, we manually added the entry for BNY.

The final table connecting each ticker symbol to its Wikidata identifier will serve as the foundation of the knowledge graph we will build. We will use the ticker symbols to help us integrate the FinReflectKG data into what we will extract from Wikidata.

Extracting Data from FinReflectKG

The source dataset, FinReflectKG, is distributed through Hugging Face as a collection of Parquet files containing millions of facts extracted from SEC filings. Rather than loading these files individually, we download the dataset locally.

Before extracting a subset, we examine the content of the data. It contains 17,513,372 extracted relationship triples. Each row represents a single relationship together with its provenance and contextual information.

Field Type Description
triplet_id string Unique identifier for the extracted relationship.
entity string Source entity participating in the relationship.
entity_type string Semantic type of the source entity (e.g., ORG, PERSON, PRODUCT).
relationship string Name of the relationship extracted from the text.
target string Target entity participating in the relationship.
target_type string Semantic type of the target entity.
start_date string Beginning of the temporal interval associated with the relationship, when available.
end_date string End of the temporal interval associated with the relationship, when available.
extraction_type string Method used to extract the relationship.
ticker string Stock ticker of the company whose SEC filing contains the extracted fact.
year int64 Reporting year of the source filing.
source_file string SEC filing from which the relationship was extracted.
page_id string Identifier of the page containing the extracted text.
chunk_id string Identifier of the text chunk used during extraction.
chunk_text string Original text from which the relationship was extracted.
triplet_length int64 Length of the extracted relationship representation.
chunk_text_length int64 Length of the source text chunk.
has_context boolean Indicates whether additional contextual information accompanies the extracted relationship.

In the table below, we show representative has_stake_in relationships in this dataset. he examples illustrate that the relationship is applied to a diverse collection of semantic categories rather than being restricted to equity ownership between organizations.

Ticker Entity Entity Type Target Target Type Observation
EL Lauder family PERSON EL ORG A representative ownership relationship between a person and an organization.
ARNC Arconic ORG CBG COMP Target is classified as a company rather than an organization, illustrating inconsistent typing.
BBY Best Buy ORG North America Retail Segment SEGMENT Business segment incorrectly appears as the target of an ownership relationship.
BAC Bank of America ORG Mortgage-backed Security FIN_INST Financial instrument is modeled as an owned entity rather than as an investment asset.
PFE Pfizer ORG Paxlovid PRODUCT Product appears as the target; a produces relationship would be more appropriate.

These observations motivated the introduction of relationship-specific type constraints during preprocessing. Rather than accepting every extracted has_stake_in relationship, we retain only those whose source is either an organization or person and whose target is an organization. This simple filtering step retains semantically clean relationships. We curate the constraints for each of the relationship of interest as shown below.

Relationship Allowed Source Types Allowed Target Types Semantic Interpretation
parent_of ORG ORG Corporate parent subsidiary relationship.
subsidiary_of ORG ORG Corporate subsidiary relationship.
produces ORG PRODUCT Organization manufactures or markets a product.
invests_in ORG, PERSON ORG Organization or person invests in another organization.
has_stake_in ORG, PERSON ORG Organization or person owns an equity stake in another organization.
supply ORG ORG Supplier customer relationship between organizations.

Applying the constraints eliminates many semantically invalid relationships, such as organizations having ownership stakes in business segments, financial instruments, or products, thereby producing a substantially cleaner knowledge graph for downstream entity reconciliation and graph loading.

The filtered results produce merely 4207 rows. It is a very small subset of the overall dataset, but enough to illustrated the challenges faced in integrating data from two different sources.

Loading Data into Neo4j

We will now load the company data derived from Wikdiata and the FinReflectKG into Neo4j to form an integrated graph.

Loading the Wikidata Information

The loading process first creates all the company nodes, and then connects them using edges.

For creating the company nodes, we leverage the curated list of companies we have created previously. We use the wikidata_id as the merge key for the company. We record the company name, and its ticker symbol as node properties.

For creating edges, we leverage the data we previously extracted. We group the Neo4j requests by property. As some of the qualifier properties are not always present, we handle them using conditionals in the Cypher query. Furthermore, as Neo4j does not allow the property names to be passed as variables, we are forced explicitly enumerate all the properties. The qualifier properties are added as edge properties.

Since each qualifier associated with a Wikidata statement is represented as a separate row in the CSV file, multiple rows may correspond to the same relationship. The relationship is therefore merged on the statement identifier, and the qualifier rows are used to populate additional properties on that relationship. This design faithfully preserves the Wikidata statement model while producing a compact property graph.

Several improvements to the loader are possible including scanning the property rows only once, generating the Cypher edge creation command programmatically, and better code organization.

Integrating FinReflectKG

FinReflectKG data contains companies and people, which also exist in Wikidata, but it does not have their Wikidata identifiers. Mapping the text identifiers for companies and people into Wikidata identifiers is the primary data integration task we address in the sample solution.

To perform the reconciliation, we use the Wikidata reonciliation service that takes input text, and produces a Wikidata identifier for it. We record the results returned by this service as two seprate CSV files -- one for companies, and hte other for people.

While loading the FinReflectKG data into Neo4j, we retain only those rows which we were able to successfuly reconcile. Even though that further limits the incremental data to the graph, but this exercise is illustrative to appreciate the complexity of merging two datasets coming from different sources