How to Query and Download Public Cancer Imaging DICOM Data with Imaging Data Commons

Table of contents

• Scientific introduction
  • Why imaging discovery starts with metadata
  • DICOM identity and semantic verification
  • Licensing and responsible reuse
  • Size estimation before transfer
  • Release and package reproducibility
• Test progress
• Demo user request
• Demo data
• Validated workflow
• Results and artifacts
• Reproducibility
• Limitations
• References
• Try this workflow

The National Cancer Institute Imaging Data Commons provides public radiology, pathology, and derived imaging data with searchable metadata and explicit licensing. A reproducible download begins with a narrow metadata query, not a bulk transfer: identify the exact series, inspect its modality, instance count, estimated size, and license, then download and reopen every DICOM file. In the verified example below, idc-index 0.12.4 queried IDC data release v24 for one presentation-state series. The selected row reported one 0.001266 MB instance under CC BY 3.0. The downloaded DICOM was reopened with pydicom 3.0.2, and its Study, Series, SOP Instance UID, and modality were preserved in machine-readable and human-readable results.

Scientific introduction

Why imaging discovery starts with metadata

Medical-imaging repositories contain data at several levels. A collection groups related subjects and studies; a patient can have multiple studies; each study can contain several series; and each series normally contains one or more DICOM instances. A chest CT acquisition, a segmentation, a structured report, and a presentation state may share a study while representing very different objects. Downloading “the study” without examining series-level metadata can therefore retrieve an inappropriate modality, derived object, unexpected volume, or data with unsuitable reuse terms.

Imaging Data Commons, commonly abbreviated IDC, organizes public cancer imaging data and makes a compact metadata index available through the idc-index Python package. The official IDC portal and documentation describe cloud-hosted data, browser visualization, and programmatic access Imaging Data Commons documentation. The package exposes an SQL query interface over release-specific index tables idc-index documentation. A query can select series by collection, modality, body part, manufacturer, patient, study, or DICOM UID before any pixel or object data are transferred.

A SeriesInstanceUID is the stable DICOM identifier for a series. It is not a filename and should be treated as a string, because its dotted numerical appearance does not imply arithmetic meaning. StudyInstanceUID groups related series, while SOPInstanceUID identifies an individual DICOM object. Series-level indexes can report Study and Series identifiers without exposing every SOP Instance UID. Consequently, the instance identifier should be obtained by reopening the downloaded object rather than invented from the series identifier.

DICOM identity and semantic verification

DICOM is both a file format and a communication standard. A DICOM object contains a dataset of tagged attributes, including unique identifiers, modality, transfer syntax, and object-class information. Some modalities contain pixels; others represent segmentations, presentation states, radiation-therapy structures, annotations, or reports. The selected positive control has modality PR, meaning it is a Grayscale Softcopy Presentation State rather than a conventional CT image. This is useful for testing because it is a real, very small public object, but it must not be described as a diagnostic image volume.

The DICOM Standard is maintained by the National Electrical Manufacturers Association and specifies information objects, services, data structures, and unique identifiers DICOM Standard. A robust validation opens the downloaded file with a DICOM-aware parser such as pydicom and reads the observed tags. File existence or a .dcm suffix proves almost nothing: an HTML error page, partial download, zero-byte object, or unrelated DICOM can all occupy the expected path.

This workflow therefore compares the reopened StudyInstanceUID, SeriesInstanceUID, and Modality with the selected index row. It records the reopened SOPInstanceUID even though the series-granularity row does not expose that field. The exact observed SOP Instance UID was 1.2.276.0.7230010.3.1.4.3496549722.492.1315596374.99. The distinction between “not available in this index row” and “missing from the DICOM” is scientifically important.

Licensing and responsible reuse

Public access does not mean every dataset has identical reuse terms. IDC metadata includes license fields because some collections permit broad reuse while others restrict commercial use or impose attribution requirements. The selected series reports CC BY 3.0. That license permits sharing and adaptation subject to attribution and other stated conditions Creative Commons Attribution 3.0. Researchers should preserve collection identifiers, source DOI, license text, and citations when redistributing data or derived results.

Licensing is separate from privacy and research governance. IDC data are de-identified public research resources, but a downstream project may combine them with private clinical data or generate sensitive inferences. Institutional policies, data-use agreements, security controls, and ethical review may still apply. A tutorial result should not imply that a technical download grants permission for every proposed use.

Size estimation before transfer

Medical-imaging collections can contain millions of instances and reach terabyte scale. A small SQL query can reveal instanceCount and series_size_MB before download. This allows an agent or researcher to stop, refine selection, or request approval before a costly transfer. The local validation policy limits required package and data transfers to less than 500,000,000 bytes. The complete planned software transfer was 255,188,610 bytes, while the selected DICOM was only 1,266 bytes.

Size fields are estimates rather than integrity proofs. The workflow checks the number of downloaded DICOM files and their nonzero sizes after transfer. Larger selections should also record checksums, retry outcomes, available disk space, and whether multiple cloud mirrors returned identical content. Parallel download settings should be bounded so a metadata mistake does not become an uncontrolled transfer.

Release and package reproducibility

idc-index includes metadata tied to an IDC release. Results can change when collections are added, corrected, or retired. A reproducible report therefore records both the Python package version and IDCClient().get_idc_version(). In this case the values were idc-index 0.12.4 and IDC v24. Pydicom 3.0.2 performed the object reopen.

The versions must come from the same isolated interpreter that runs the query. Copying versions from documentation, a global Python installation, or an earlier attempt could conceal environment drift. The installed skill provides a wrapper that enters its retained environment for version checks, metadata queries, public download attempts, and DICOM validation.

Test progress

Several retained attempts improved the delivered workflow. A direct chat download could not reach either official cloud mirror because the sandbox restricted object-store networking. The native lifecycle had already downloaded the exact official file successfully, so the workflow added a checksum-pinned official cache as a fallback. The cache can be used only after public-mirror failure, and it must be reopened and matched against live index metadata.

Later attempts exposed reporting drift: one report omitted the displayed SOP Instance UID, another changed JSON key names, and another failed to use the installed interpreter. The instructions now require a stable result schema, literal identifier and checksum values, and package execution through the skill-owned wrapper. These were delivery defects, so the fixes were written into the actual skill rather than hidden in test code.

Demo user request

Use Imaging Data Commons to inspect the selected series UID in data/selection.json. Query its current metadata and license, estimate the download size, then download the series from the public IDC store. Save the complete row as outputs/metadata.csv, the native DICOM as outputs/dicom/selected.dcm, a readable outputs/summary.md, and machine-readable outputs/results.json. Reopen the DICOM with pydicom and report the actual Study, Series, SOP Instance UID and modality. Attempt public mirrors first; only if the chat sandbox cannot reach them, verify and use the official cached file and its SHA-256.

The request specifies a scientific endpoint, input manifest, validation obligations, and deliverable contract. It does not contain a prewritten analysis program. The agent must load the installed instructions, query the release-specific index, inspect licensing and size, handle the allowed network fallback, and verify the resulting object.

Demo data

The tracked selection manifest contains the chosen SeriesInstanceUID and expected public metadata. It identifies one official NCI IDC series from collection qiba_ct_1c. The manifest is small enough to audit directly and contains no private clinical data. It is a selection request, not a fabricated metadata response.

The retained official cache is not a generic fixture. It is the exact file produced by the passing native IDC download. Its SHA-256 is 201fd8af201dbd48aff60960dc386f2e41141f20f5bb468b6a143ecda63de804. A provenance JSON accompanies it during chat execution and limits its role to network-failure fallback.

Validated workflow

The workflow opened data/selection.json, treated the series UID as a string, and queried the IDC index for the complete selected row. It confirmed one match, captured license and size fields, generated a browser viewer URL, and attempted the official download. When the sandbox could not resolve either public mirror, it copied only the retained official file whose checksum matched the attached provenance.

The downloaded or cached object was reopened with pydicom inside the same isolated environment. The observed Study and Series identifiers and modality were compared with the index. The SOP Instance UID was preserved as an observed DICOM value, with an explicit explanation that the series-level index did not expose it.

from idc_index import IDCClient
import pydicom

client = IDCClient()
row = client.sql_query(
    "SELECT * FROM index WHERE SeriesInstanceUID = '<selected UID>'"
)
client.download_from_selection(
    seriesInstanceUID=["<selected UID>"],
    downloadDir="outputs/public_download",
)
dataset = pydicom.dcmread("outputs/dicom/selected.dcm")

This code is transparent reproduction information, not a substitute for the conversational request or semantic checks.

Results and artifacts

The complete metadata CSV, machine-readable verification, readable summary, and retained native DICOM form the primary deliverables.

The screenshot focuses on the rendered result rather than a file explorer or raw JSON editor. The two data-derived images expose result fields and retained file sizes. Their sources and hashes are recorded in publication manifests.

Reproducibility

Validation completed on 2026-07-27 using Ubuntu 22.04.5 LTS, Linux x86_64 CPU, idc-index 0.12.4, IDC v24, and pydicom 3.0.2. CUDA was not required. The isolated package environment, official download cache, input manifest, generated artifacts, and failure evidence remain available to developers.

Reproduction requires the exact selection manifest, package versions, IDC release, SQL query, license field, viewer URL, official-cache checksum when used, and all observed DICOM identifiers. A later IDC release should be treated as a new validation because metadata, collection membership, or availability can change.

Limitations

This test validates one tiny presentation-state series, not a CT or MR volume and not a pathology slide. It does not test pixel-array analysis, volumetric geometry, segmentations, annotations, clinical-table joins, bulk cohort downloads, DICOMweb, or authenticated BigQuery. It is a deterministic access and identity test.

The public mirrors were reachable from the native lifecycle but not from the sandboxed conversational process. The verified cache demonstrates a safe recovery path; it does not prove that every deployment can reach IDC object storage. Production systems should test their network, proxies, TLS trust, disk capacity, and cloud-mirror policies.

No clinical or diagnostic claim follows from this object. The PR modality is a presentation state. License compliance, dataset citation, privacy review, representative sampling, and scientifically appropriate analysis remain the responsibility of each downstream project.

References

1. National Cancer Institute. Imaging Data Commons documentation.
2. Imaging Data Commons. idc-index documentation.
3. National Electrical Manufacturers Association. DICOM Standard.
4. Creative Commons. Attribution 3.0 Unported.

Try this workflow

MindPlot has built-in support for the demonstrated Imaging Data Commons skill. Users can provide a natural-language request and selection data while the MindPlot agent writes and runs the query, verifies the license and DICOM identity, and returns the deliverables without requiring users to write the code shown above. Try it at mindplot.ai, or download the desktop version for a better experience and stronger local-data privacy.