Add module to download-upload alpha earth embeddings.

[manugv]

Added module to download alpha earth embeddings, convert them to COGs and upload them to S3 bucket.

[Copilot]

Pull request overview

Adds a new alphaearth_utils utility module to support an AlphaEarth embeddings ingestion workflow (select AOI tiles → download embeddings → convert to COG → upload to S3) within the eo_processing.utils package.

Changes:

• Introduces grid/AOI intersection helpers to identify relevant embedding tiles.
• Adds download helpers with S3 (unsigned) and HTTP fallback.
• Adds VRT patching, GDAL-based COG translation, and S3 upload/filter helpers.

Suppressed comments (7)

src/eo_processing/utils/alphaearth_utils.py:56

• Calling logging.basicConfig(...) at import time changes global logging configuration for any application importing this module. Library modules should generally only define a logger and leave configuration to the application entrypoint.

# Configure logging
logging.basicConfig(
    level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s"
)
logger = logging.getLogger(__name__)

src/eo_processing/utils/alphaearth_utils.py:232

• get_intersecting_grids performs spatial operations without ensuring grid_gdf is in the same CRS as input_gdf. If the CRSs differ, intersections will be incorrect (and GeoPandas typically warns about it). Reproject the grid to input_gdf.crs before building the spatial index / running intersects.

    grid_gdf = gpd.read_file(grid_file)
    input_union = input_gdf.union_all()

src/eo_processing/utils/alphaearth_utils.py:151

• download_s3_file warns when the companion .vrt does not exist, but then still calls download_file for it unconditionally, which will raise and cause the whole download to be reported as failed. Only attempt to download the VRT when it exists (or handle the exception separately).

        if not self.s3_object_exists(bucket, key1):
            logger.warning(f"Object does not exist: s3://{bucket}/{key1}")

        try:
            output_path = Path(output_dir) / Path(key)

src/eo_processing/utils/alphaearth_utils.py:388

• translate_to_cog builds a shell command string and runs it with shell=True, which is unsafe (path escaping/injection) and fragile for filenames with spaces. Prefer subprocess.run([...], check=True) with proper argument splitting, and use the module logger instead of print.

    path_out = path_vrt.parent / f"{path_vrt.stem}_{version}.tif"

    if path_out.exists():
        print(f"File {path_out} already exists, skipping.")
        return path_out

    gdal_cmd = " ".join(
        ["gdal_translate"] + GDAL_COG_OPTIONS + [str(path_vrt), str(path_out)]
    )
    try:
        subprocess.check_call(gdal_cmd, shell=True)

src/eo_processing/utils/alphaearth_utils.py:480

• filter_files_on_s3 parses S3 keys by fixed indices (split("/")[1], etc.). get_s3_content(s3_root) returns keys that include the prefix, so these indices will shift and the extracted year/zone/filename will be wrong. Strip s3_root from the key before parsing (and guard against unexpected key formats).

    list_all = storage.get_s3_content(s3_root)
    all_files = [f["Key"] for f in list_all if f["Key"].endswith(".tif")]
    # Build a list of dicts as before
    result = [
        {"year": f.split("/")[1], "zone": f.split("/")[2], "filename": f.split("/")[3].split("_")[0]}
        for f in all_files
    ]

src/eo_processing/utils/alphaearth_utils.py:330

• overwrite is documented for download_embeddings, but it is only honored in the HTTP branch. In the S3 branch, existing local files will always be re-downloaded/overwritten. Add a local existence check when overwrite=False to keep behavior consistent.

        for fl in filenames:
            success, new_path = client.download_s3_file(fl, output_dir)
            if success:
                new_paths.append(new_path)
                successful_downloads += 1

src/eo_processing/utils/alphaearth_utils.py:189

• The HTTP fallback downloads only the .tiff file. The S3 path downloads both .tiff and its companion .vrt, and later processing (Path(f).with_suffix(".vrt")) assumes the VRT exists locally. The HTTP download should also fetch the corresponding .vrt (and should set a timeout to avoid hanging indefinitely).

        url = fl.replace(S3_BASE_URL, "https://data.source.coop/")
        out_path = final_dir / fl_path.name
        if out_path.exists() and not overwrite:
            logger.info(f"File {out_path} exists, skipping download.")
            new_paths.append(str(out_path))

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.