GCSFileChangeStream is a source primitive that can be used to subscribe to file changes on a given GCS bucket. To create a GCSFileChangeStream provide:

ArgDescription
gcs_bucket GCSBucket to subscribe to change events from. NOTE: you do not need to provide file path or file format.
event_typesList of event types to subscribe to. Defaults to: [GCSChangeStreamEventType.OBJECT_FINALIZE]

You must mark your GCSFileChangeStream primitive as managed to have the underlying resources created.


from buildflow.io.gcp import GCSBucket, GCSFileChangeStream
from buildflow.types.gcp import GCSFileChangeEvent

bucket = GCSBucket("project", bucket_name="my-bucket")
source = GCSFileChangeStream(gcs_bucket=bucket)

app.manage(bucket, source)

@app.consumer(source=source, sink=...)
def my_consumer(elem: GCSFileChangeEvent):
    ...

With event types:

from buildflow.io.gcp import GCSBucket, GCSFileChangeStream
from buildflow.types.gcp import GCSFileChangeEvent, GCSChangeStreamEventType

bucket = GCSBucket("project", bucket_name="my-bucket")
source = GCSFileChangeStream(
    gcs_bucket=bucket,
    event_types=[GCSChangeStreamEventType.OBJECT_FINALIZE, GCSChangeStreamEventType.OBJECT_DELETE],
)

app.manage(bucket, source)

@app.consumer(source=source, sink=...)
def my_consumer(elem: GCSFileChangeEvent):
    ...

​
Types

The GCSFileChangeStream source always returns a GCSFileChangeEvent object. This object contains the following fields:

FieldDescription
event_typeThe buildflow.types.gcp.GCSChangeStreamEventType event type
metadataAny metadata that was associated with the file change event. This is a dictionary of strings.
blobThe raw contents of the file. This is lazily fetched.

​
Resource Creation

If you are using BuildFlow’s built in resource creation/management you can use the GCSFileChangeStream primitive type to create and manage all resources that are need to subscribe to a GCS bucket.

The following resources will be created:

  • GCP PubSub Topic
  • GCP PubSub Subscription
  • GCP Bucket Notification
  • GCS Bucket
  • IAM permissions for GCS service account
Pub/SubFile