Skip to content

Add GCS download mode for Tennessee diabetes import - #2159

Merged
rohitkumarbhagat merged 3 commits into
datacommonsorg:masterfrom
rohitkumarbhagat:tennessee-diabetes-gcs
Aug 14, 2026
Merged

Add GCS download mode for Tennessee diabetes import#2159
rohitkumarbhagat merged 3 commits into
datacommonsorg:masterfrom
rohitkumarbhagat:tennessee-diabetes-gcs

Conversation

@rohitkumarbhagat

@rohitkumarbhagat rohitkumarbhagat commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add --download_source=tn|gcs to the Tennessee diabetes downloader.
  • Keep TN.gov as the default for direct execution.
  • Enable GCS mode in the import manifest. - last download from TN.gov passed. Keeping it default.
  • Generate yearly filenames from 2019 through the current year for both modes.
  • Skip missing GCS years, fail when no files are downloaded, and propagate unexpected GCS errors.
  • Log download source, destination, completion, and file size.
  • Document the semi-manual GCS staging workflow.

Motivation

TN.gov blocks requests originating from Google Cloud environments. GCS staging provides a semi-manual fallback while preserving the existing processing flow.

Validation

  • Added four focused unit tests for URL generation and GCS behavior.
  • All focused tests pass.
  • Repository lint passes.
  • Completed a live GCS download and Excel-processing validation for the 2019–2022 files.

The runtime service account requires storage.objects.get access to the unresolved_mcf bucket.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a semi-manual GCS fallback mode to the Tennessee diabetes data importer, allowing files to be downloaded from Google Cloud Storage when TN.gov is blocked. It updates the README, adds a --download_source flag and GCS download logic to download.py, and introduces a new test suite in download_test.py. The review feedback correctly points out a potential compatibility issue with e.add_note(), which requires Python 3.11+ and could cause runtime errors in older environments.

Comment thread statvar_imports/nyu_diabetes/tennessee/download.py
Comment thread statvar_imports/nyu_diabetes/tennessee/download.py
Comment thread statvar_imports/nyu_diabetes/tennessee/download.py
@rohitkumarbhagat
rohitkumarbhagat enabled auto-merge (squash) August 14, 2026 14:53
@rohitkumarbhagat
rohitkumarbhagat merged commit 06fcc4b into datacommonsorg:master Aug 14, 2026
14 checks passed
@rohitkumarbhagat
rohitkumarbhagat deleted the tennessee-diabetes-gcs branch August 14, 2026 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants