Google Cloud Storage

Google Cloud Storage works with Beam through its XML API, which is compatible with the Amazon S3 API. Use the SDK's S3-compatible config with the endpoint https://storage.googleapis.com and an HMAC key. As with every connector, the key stays in your process: workers receive only the short-lived access the SDK signs for their chunk.

Create an HMAC key

1. In the Google Cloud console, create a service account for Beam under IAM & Admin → Service Accounts. It needs no project role.
2. On each bucket Beam reads or writes, open Permissions → Grant access and give the service account the Storage Object User role. It covers reading, writing, listing and deleting objects, and multipart uploads.
3. Under Cloud Storage → Settings → Interoperability, create a key for the service account. Google shows the secret only once.

Do not restrict the bucket to IP addresses or networks: independent workers read and write from their own addresses.

TypeScript

import { BeamClient, S3CompatibleProviderConfig, S3ProviderConfig } from "@beam-network/sdk";

const beam = new BeamClient({ apiKey: process.env.BEAM_API_KEY! });

const transfer = await beam.createTransfer({
  sources: [
    S3ProviderConfig.create({
      bucket: "aws-source",
      key: "exports/data.parquet",
      region: "us-east-1",
      access_key_id: process.env.AWS_ACCESS_KEY_ID!,
      secret_access_key: process.env.AWS_SECRET_ACCESS_KEY!,
    }),
  ],
  destinations: [
    S3CompatibleProviderConfig.create({
      provider: "gcs",
      bucket: "gcs-dest",
      key: "imports/data.parquet",
      endpoint_url: "https://storage.googleapis.com",
      access_key_id: process.env.GCS_HMAC_ACCESS_ID!,
      secret_access_key: process.env.GCS_HMAC_SECRET!,
    }),
  ],
  name: "s3-to-gcs",
});

console.log(transfer.transfer_id);

Python

Install the SDK with its S3 extra: pip install "beam-network-sdk[s3]".

from beam_network_sdk.models import (
    S3CompatibleProviderDestination,
    S3CompatibleProviderSource,
)

source = S3CompatibleProviderSource(
    provider="gcs",
    bucket="gcs-source",
    key="exports/file.bin",
    endpoint_url="https://storage.googleapis.com",
    access_key_id="...",
    secret_access_key="...",
)

destination = S3CompatibleProviderDestination(
    provider="gcs",
    bucket="gcs-dest",
    key="imports/file.bin",
    endpoint_url="https://storage.googleapis.com",
    access_key_id="...",
    secret_access_key="...",
)

Use these S3-compatible models. The Python SDK's GCSProviderSource and GCSProviderDestination are not this path: a transfer that uses them fails with NotImplementedError before any data moves.

In Studio

Studio v1.0.25 and later has a Google Cloud Storage connection. Open Credentials → New credential → Object storage → Google Cloud Storage, enter the HMAC access key ID and secret, and add the bucket so Studio can test the connection. Studio sets the endpoint itself. Then choose the connection in a source or destination storage endpoint, as for any S3-compatible bucket.

An older Studio has no such connection: use Credentials → New credential → Custom → Custom S3-compatible with Endpoint URL set to https://storage.googleapis.com.

What to expect

• Sources and destinations. A bucket can be either, in a fan-out with other destinations too.
• Transfers to Box. Box needs the SHA-1 of a file of 20 MiB or more, and Google Cloud Storage keeps none for an object, so independent operators read the source again to compute it. See Large files need their SHA-1.
• Region. Google Cloud Storage accepts any signing region; the SDK's default, us-east-1, works.