BiomarkerKB Wiki - Recent changes [en]

Data Submission/Data Upload

MariaKim — Thu, 07 May 2026 22:18:27 GMT

Standardized and Controlled Vocabulary

← Older revision		Revision as of 22:18, 7 May 2026
(2 intermediate revisions by the same user not shown)
Line 9:		Line 9:
	## <code>component_group</code> containing integers (1, 2, 3...). A multicomponent biomarker must have the same integer in all rows related to that biomarker.		## <code>component_group</code> containing integers (1, 2, 3...). A multicomponent biomarker must have the same integer in all rows related to that biomarker.
	# Other fields and annotations may also be collected from the data source, however if data is missing it can also be inferred or mapped from other sources.		# Other fields and annotations may also be collected from the data source, however if data is missing it can also be inferred or mapped from other sources.
			## <code>evidence</code> is one or more exact citations from the evidence source (in most cases, it will be the PubMed publication).
	# Apply the following standards to the data when possible:		# Apply the following standards to the data when possible:
	## <code>condition_id</code> = <code>DOID:0080600</code>		## <code>condition_id</code> = <code>DOID:0080600</code>. Refer to https://disease-ontology.org/do/.
	## <code>specimen_id</code> = <code>UBERON:0000178</code>		## <code>specimen_id</code> = <code>UBERON:0000178</code>. Refer to https://www.ebi.ac.uk/ols4/ontologies/uberon.
			## <code>loinc_code</code> = <code>LOINC:100153-6</code>. Refer to https://loinc.org/ (you may need to create an account to access the search functionality).
	## <code>evidence_source</code> = <code>SOURCE:ID</code>, for example <code>PubMed:32677844</code>		## <code>evidence_source</code> = <code>SOURCE:ID</code>, for example <code>PubMed:32677844</code>
	## For <code>assessed_biomarker_entity_id</code> please refer to ~~this~~ GitHub documentation for which standards to follow		## For <code>assessed_biomarker_entity_id</code> please refer to the [https://github.com/clinical-biomarkers/biomarker-controlled-vocabulary GitHub documentation] for which standards to follow
	# Provide extra annotations from your DCC/data with the agreed upon standards from the Biomarker Annotation RFC. This data does not have to follow the data model and can be submitted in a separate file.		# Provide extra annotations from your DCC/data with the agreed upon standards from the Biomarker Annotation RFC. This data does not have to follow the data model and can be submitted in a separate file.
	## For example: Relevant EHR data/LOINC data for biomarkers/biomarker entities can be included in a separate sheet.		## For example: Relevant EHR data/LOINC data for biomarkers/biomarker entities can be included in a separate sheet.
	# Create a tsv/json file with the agreed upon fields which correspond to the biomarker data model. The data dictionary provides details on what the different fields represent.		# Create a tsv/json file with the agreed upon fields which correspond to the biomarker data model. The data dictionary provides details on what the different fields represent.
	## The preferred method for data submission is a json file as it will help ingest the data into the existing data efficiently. However, tsv file submissions are ok as well. In the GitHub, <code>data_conversion.py</code> script exists in the Data Conversion Folder and it will handle tsv to json file conversion and json to tsv file conversion as well.		## The preferred method for data submission is a json file as it will help ingest the data into the existing data efficiently. However, tsv file submissions are ok as well. In the GitHub, <code>data_conversion.py</code> script exists in the Data Conversion Folder and it will handle tsv to json file conversion and json to tsv file conversion as well.
	## The ~~biomarker~~ data page has examples of tsv data submissions and how the data should be formatted with the appropriate biomarker fields. Example		## The [BiomarkerKB data page] has examples of tsv data submissions and how the data should be formatted with the appropriate biomarker fields. Example
	# For panel biomarkers, if the biomarkers are part of the same panel, the biomarker_id value for each biomarker should be any string value that can uniquely identify which rows are part of the same biomarker panel. Documentation		# For panel biomarkers, if the biomarkers are part of the same panel, the biomarker_id value for each biomarker should be any string value that can uniquely identify which rows are part of the same biomarker panel. Documentation
	# If curating data in tsv format: If biomarker rows are part of the same biomarker entry but differ on specimen, evidence, or role, then the biomarker_id for each row should be any string value that can uniquely identify which rows are part of the same biomarker.		# If curating data in tsv format: If biomarker rows are part of the same biomarker entry but differ on specimen, evidence, or role, then the biomarker_id for each row should be any string value that can uniquely identify which rows are part of the same biomarker.
Line 31:		Line 33:
	There is a standard way to report some biomarker data. This section covers how the actual biomarker should be reported and how other fields should be filled out.		There is a standard way to report some biomarker data. This section covers how the actual biomarker should be reported and how other fields should be filled out.

	=== ~~Condition~~ ===		=== condition ===
	~~Condition~~ should be reported in all lowercase and ~~condition ID~~ (from Disease Ontology ID) should be provided in the following column		<code>condition</code> should be reported in all lowercase and <code>condition_id</code> (from Disease Ontology, MONDO, or SNOMED) should be provided in the following column

	=== assessed_biomarker_entity ===		=== assessed_biomarker_entity ===
Line 48:		Line 50:

	=== best_biomarker_role ===		=== best_biomarker_role ===
	Report in all lowercase. Refer to the [https://www.ncbi.nlm.nih.gov/books/NBK326791/ BEST Resource] ~~for~~ the correct biomarker role.		Report in all lowercase. Refer to the [https://www.ncbi.nlm.nih.gov/books/NBK326791/ BEST Resource] to infer the correct biomarker role. Accepted role terms are:
			* diagnostic
			* monitoring
			* predictive
			* prognostic
			* response
			* risk
			* safety

	=== specimen ===		=== specimen ===
	Report in all lowercase and ~~specimen_ID~~ in the following column should be from UBERON.		Report in all lowercase and <code>specimen_id</code> in the following column should be from UBERON.

	=== biomarker ===		=== biomarker ===

Data Release Notes

MariaKim — Wed, 29 Apr 2026 15:27:03 GMT

← Older revision		Revision as of 15:27, 29 April 2026
Line 5:		Line 5:
	* The third digit (Z) is updated for bug fixes or minor changes.		* The third digit (Z) is updated for bug fixes or minor changes.

	== Version 2.4.3 ~~(draft)~~ ==		== Version 3.2.1 ==
			Date: April 23rd, 2026
			=== Data Updates ===
			* Added a new dataset containing user-submitted biomarkers.
			* Added a new manually curated dataset of biomarkers with exposure agents.

			== Version 3.1.1 ==
			Date: April 9th, 2026
			=== Backend Updates ===
			Complete overhaul of the backend data pipeline architecture:
			* Improved ETL processes for greater reliability and scalability
			* Enhanced data validation and error handling across pipeline stages
			* Optimized performance for faster data processing and reduced runtime
			* Refactored codebase for maintainability and extensibility

			== Version 2.4.3 ==
	Date: February 26th, 2026		Date: February 26th, 2026
	=== Data Updates ===		=== Data Updates ===
	* An image biomarker ~~was added to~~ OncoMX ~~data~~.		* Added a new image-based biomarker from the OncoMX dataset.
	* ~~Fix UPKB~~ biomarkers ~~containing~~ exposure agents.		* Fixed UniProtKB biomarkers that incorrectly included exposure agents.
	=== Backend Updates ===		=== Backend Updates ===
	* ~~Read~~ ChEBI API ~~response which is in~~ JSON ~~format~~		* Updated ChEBI API integration to properly parse JSON responses.

	== Version 2.4.2 ==		== Version 2.4.2 ==

BiomarkerKB Data Processing and Modeling Specification

MariaKim — Tue, 28 Apr 2026 20:07:16 GMT

Created page with "= Biomarker identification = A biomarker’s canonical identity is defined by removing the disease or exposure agent dimension from its component combination. For each row in the source TSV, a combination list is first constructed out of three components: (1) assessed entity identifier, (2) condition or exposure agent identifier, and (3) controlled vocabulary term. The condition or exposure agent component is then removed to generate a canonical combination consisting on..."

New page

= Biomarker identification =
A biomarker’s canonical identity is defined by removing the disease or exposure agent dimension from its component combination. For each row in the source TSV, a combination list is first constructed out of three components: (1) assessed entity identifier, (2) condition or exposure agent identifier, and (3) controlled vocabulary term. The condition or exposure agent component is then removed to generate a canonical combination consisting only of assessed entity identifier and controlled vocabulary pairs. This canonical representation is hashed using MD5 to produce a stable identifier, which serves as a condition-independent anchor across datasets.

Second-level biomarker identifiers are assigned by appending an index to the canonical ID, where the index increments for each distinct condition or exposure agent linked to the same canonical combination, resulting in identifiers such as AN6278-1 and AN6278-2.

To ensure persistence across releases, previously assigned canonical IDs are reused by scanning historical ID-tracking directories. New canonical entries receive sequential identifiers (e.g., BMKB000270). Changes over time are recorded through history tracking, capturing relationships such as inheritance, replacement, and discontinuation.

= Deduplication =
Deduplication is performed at both the biomarker component and row levels to ensure that identical biomarkers are consistently identified across sources. At the component level, the fundamental unit of deduplication is a normalized combination key, defined as a sorted and JSON-serialized list of assessed entity ID, condition or exposure agent ID, and controlled vocabulary tuples. Records from different sources that resolve to the same lowercased combination key are treated as the same biomarker and assigned a shared identifier. Provenance is preserved through a mapping from each unique combination key to the set of source files that contributed it, allowing aggregation without duplication.

At the row level, exact duplicate entries within individual TSV files are removed during ingestion using a dictionary keyed by the serialized row content.

A critical prerequisite for effective deduplication is controlled vocabulary normalization. Semantically equivalent biomarker descriptions, such as “increased IL6 level” and “elevated IL-6 levels,” must first be standardized to a common controlled vocabulary term; otherwise, their component combinations will not match and will be treated as distinct entities.

= Data Modeling =
Data modeling begins with controlled vocabulary normalization. Raw biomarker strings are tokenized and matched against defined pattern definitions, where each rule maps to a structured label of the form change_type, aspect_type, and mod_type (for example, increased, level, not_specified). The matched rule is then used to generate a standardized representation such as “Increased level of protein IL6/UPKB:P05231.” Special cases, including single-nucleotide polymorphisms, mutations, and glycan modifications, are handled through explicit logic. Terms that do not match any pattern are flagged as “[biomarker_term_in_review],” while a supplementary rules file captures hard-coded overrides for edge cases.

The normalized data are then assembled into a document-oriented model, where each record in the c_biomarker MongoDB collection represents a single biomarker entry. Each document includes core biomarker identifiers (canonical IDs and second-level IDs) and a biomarker component array describing each assessed entity, including its identifier, type, controlled vocabulary term, associated specimens, and supporting evidence sources. Disease context is captured in a condition object containing standardized names and synonyms derived from a disease database. Additional fields capture biomarker roles (such as diagnostic or prognostic), aggregated evidence sources, and citation metadata sourced from PubMed. Where available, documents also include normal range statistics, a list of contributing upstream sources, and cross-references to external databases.

= ETL (Extract, Transform, Load) Processes =
The ETL pipeline is organized as a staged workflow that transforms raw source data into structured, queryable databases. In the first step, literature ingestion is performed by a set of scripts, which collect all PubMed identifiers referenced in the source TSV files, retrieve the corresponding MEDLINE XML records, and extract structured citation data into JSON files.

The second step focuses on building reference databases. Disease objects are constructed from GlyGen and Disease Ontology sources, and statistical summaries are computed, including minimum, maximum, mean, median, interquartile range, and whiskers, stratified by age group and sex using clinical datasets from Oracle Health and GWDC.

In the third step, the core databases are assembled. The main biomarker collection is generated, integrating normalized biomarker records with disease annotations, citation data, normal ranges, and cross-references. Supporting structures are then created for efficient querying by generating flattened list representations optimized for search results, including computed relevance scores and filter bitmaps; all searchable fields are tokenized into phrase-level indices to support the search layer.

The final step produces auxiliary databases. Aggregate statistics are generated, initialization metadata for the search system is written, and precomputed sort orders for list fields are created.

All stages of this pipeline depend on upstream dataset preparation. The preprocessing step ingests raw TSV files, applies controlled vocabulary normalization, assigns component group identifiers to link multi-component biomarkers, and outputs intermediate TSV files that serve as the input to the downstream object construction pipeline.

Data Submission/Data Upload

MariaKim — Tue, 21 Apr 2026 18:58:49 GMT

Instructions to submit Biomarker Data

← Older revision		Revision as of 18:58, 21 April 2026
(2 intermediate revisions by the same user not shown)
Line 4:		Line 4:
	# Biomarker data collected should follow the biomarker data model.		# Biomarker data collected should follow the biomarker data model.
	# "Core" fields should be filled in from the data source where biomarker data is collected. Core fields:		# "Core" fields should be filled in from the data source where biomarker data is collected. Core fields:
	## biomarker		## <code>biomarker</code>
	## assessed_biomarker_entity and assessed_biomarker_entity_id		## <code>assessed_biomarker_entity</code> and <code>assessed_biomarker_entity_id</code>
	## condition and condition_id OR exposure_agent and exposure_agent_id		## <code>condition</code> and <code>condition_id</code> OR <code>exposure_agent</code> and <code>exposure_agent_id</code>
			## <code>component_group</code> containing integers (1, 2, 3...). A multicomponent biomarker must have the same integer in all rows related to that biomarker.
	# Other fields and annotations may also be collected from the data source, however if data is missing it can also be inferred or mapped from other sources.		# Other fields and annotations may also be collected from the data source, however if data is missing it can also be inferred or mapped from other sources.
	# Apply the following standards to the data when possible:		# Apply the following standards to the data when possible:
	## condition_id = DOID		## <code>condition_id</code> = <code>DOID:0080600</code>
	## specimen_id = UBERON		## <code>specimen_id</code> = <code>UBERON:0000178</code>
	## evidence_source = "SOURCE":"ID"		## <code>evidence_source</code> = <code>SOURCE:ID</code>, for example <code>PubMed:32677844</code>
	## For assessed_biomarker_entity_id please refer to this GitHub documentation for which standards to follow		## For <code>assessed_biomarker_entity_id</code> please refer to this GitHub documentation for which standards to follow
	# Provide extra annotations from your DCC/data with the agreed upon standards from the Biomarker Annotation RFC. This data does not have to follow the data model and can be submitted in a separate file.		# Provide extra annotations from your DCC/data with the agreed upon standards from the Biomarker Annotation RFC. This data does not have to follow the data model and can be submitted in a separate file.
	## For example: Relevant EHR data/LOINC data for biomarkers/biomarker entities can be included in a separate sheet.		## For example: Relevant EHR data/LOINC data for biomarkers/biomarker entities can be included in a separate sheet.
	# Create a tsv/json file with the agreed upon fields which correspond to the biomarker data model. The data dictionary provides details on what the different fields represent.		# Create a tsv/json file with the agreed upon fields which correspond to the biomarker data model. The data dictionary provides details on what the different fields represent.
	## The preferred method for data submission is a json file as it will help ingest the data into the existing data efficiently. However, tsv file submissions are ok as well. In the GitHub, data_conversion.py script exists in the Data Conversion Folder and it will handle tsv to json file conversion and json to tsv file conversion as well.		## The preferred method for data submission is a json file as it will help ingest the data into the existing data efficiently. However, tsv file submissions are ok as well. In the GitHub, <code>data_conversion.py</code> script exists in the Data Conversion Folder and it will handle tsv to json file conversion and json to tsv file conversion as well.
	## The biomarker data page has examples of tsv data submissions and how the data should be formatted with the appropriate biomarker fields. Example		## The biomarker data page has examples of tsv data submissions and how the data should be formatted with the appropriate biomarker fields. Example
	# For panel biomarkers, if the biomarkers are part of the same panel, the biomarker_id value for each biomarker should be any string value that can uniquely identify which rows are part of the same biomarker panel. Documentation		# For panel biomarkers, if the biomarkers are part of the same panel, the biomarker_id value for each biomarker should be any string value that can uniquely identify which rows are part of the same biomarker panel. Documentation
	# If curating data in tsv format: If biomarker rows are part of the same biomarker entry but differ on specimen, evidence, or role, then the biomarker_id for each row should be any string value that can uniquely identify which rows are part of the same biomarker.		# If curating data in tsv format: If biomarker rows are part of the same biomarker entry but differ on specimen, evidence, or role, then the biomarker_id for each row should be any string value that can uniquely identify which rows are part of the same biomarker.

	=== Once data is formatted and cleaned please send any data to ~~daniallmasood~~@gwu.edu ~~===~~		=== Submission ===
			Once data is formatted and cleaned please send any data to mazumder_lab@gwu.edu.
	# Concurrently with submitting data please fill out the BCO Information: Biomarker Data Google Form.		# Concurrently with submitting data please fill out the BCO Information: Biomarker Data Google Form.
	## This will give metadata and description on how biomarker data was collected and is important for adding submitted data to the Biomarker Data page. An example of a previous BCO is provided in the sheet and available on the biomarker data page as well. [https://hivelab.biochemistry.gwu.edu/biomarker-partnership/data/BCO_000435 Example]		## This will give metadata and description on how biomarker data was collected and is important for adding submitted data to the Biomarker Data page. An example of a previous BCO is provided in the sheet and available on the biomarker data page as well. [https://hivelab.biochemistry.gwu.edu/biomarker-partnership/data/BCO_000435 Example]

Biomaker Ontology Working Group

MariaKim — Tue, 21 Apr 2026 15:20:39 GMT

MariaKim moved page Biomaker Ontology Working Group to Biomarker Ontology Working Group Misspelled title

New page

The Biomarker Ontology Working Group is a collaborative effort dedicated to developing, refining, and maintaining standardized ontological frameworks that support the representation, integration, and interoperability of biomarker-related data. The group brings together ontology experts, data curators, biomarker experts, to establish consistent terminologies, formal relationships, and semantic structures that enhance the discoverability and reuse of biomarker information across diverse biomedical resources. By aligning with community standards and promoting transparent, open development practices, the working group plays a central role in ensuring that biomarker data can be reliably linked, compared, and analyzed within translational research, clinical informatics, and computational biology applications.

The Biomarker Ontology Working Group have jointly developed '''Ontology for Biomarkers of Clinical Interest (OBCI)'''. A fully functioning ontology in OWL format can be found at https://proteininformationresource.org/staff/nataled/OBCI/core/obci_core_full.owl

The Ontology Working Group has also created a '''Controlled Vocabularies (CV)''' for standardizing the representation of the biomarker values and the biomarker entity types. The CV can be found at [[Controlled Vocabulary and Keywords]]

Biomarker Ontology Working Group currently contains the members:
{| class="wikitable"
|'''Name'''
|'''ORCID'''
|-
|'''Darren A. Natale'''
|0000-0001-5809-9523
|-
|'''Raja Mazumder'''
|0000-0001-8823-9945
|-
|'''Daniall Masood'''
|0000-0001-7441-1628
|-
|'''Hande Küçük McGinty'''
|0000-0002-9025-5538
|-
|'''Marc E. Gillespie'''
|0000-0002-5766-1702
|-
|'''Astghik Sargsyan'''
|0000-0002-5860-6369
|-
|'''Soheil Abadifard'''
|0000-0002-2980-4251
|-
|'''Jeet K. Vora'''
|0000-0002-5317-1458
|}

BiomarkerKB Resource Integration

MariaKim — Sat, 18 Apr 2026 02:37:29 GMT

← Older revision		Revision as of 02:37, 18 April 2026
(2 intermediate revisions by the same user not shown)
Line 2:		Line 2:

	= Resources for Exploration =		= Resources for Exploration =
	*[https://cadsr.cancer.gov/onedata/Home.jsp CADSR Cancer]
	*[https://themarker.idrblab.cn/ Marker Database]		*[https://themarker.idrblab.cn/ Marker Database]
	*biomarker.org
	*ResMarkerDB		*ResMarkerDB
	*SalivaDB		*SalivaDB
Line 167:		Line 165:
	= Pending Resources =		= Pending Resources =
	== biomarker.org ==		== biomarker.org ==
	~~Sent a message~~ on March 17th, 2026 ~~via~~ biomarker.~~org contact form~~, ~~followed up March 24th~~, ~~2026~~. ~~Awaiting response~~.		Reached out on March 17th, 2026 regarding data access and sent follow-up communications; however, no response was received.

			== [https://cadsr.cancer.gov/onedata/Home.jsp caDSR] ==
			The Cancer Data Standards Registry and Repository (caDSR) is a metadata registry, not a biomarker knowledge source. It defines Common Data Elements (CDEs), including field names, definitions, and controlled value sets, but does not contain biomarker-condition relationships or evidence.

			For BiomarkerKB, it could potentially be valuable for schema standardization rather than data ingestion. For example, align fields like condition, specimen, and entity type to controlled vocabularies (via NCI Thesaurus).

			Recommendation: not ingestible as a data source.