Forem: Yoshiki Fujiwara(藤原善基)@AWS Community Builder

Near-Real-Time Processing, ML Inference, and Observability for FSx for ONTAP S3 Access Points — Phase 3 Architecture Patterns

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Wed, 06 May 2026 11:02:17 +0000

TL;DR

This is Phase 3 of the FSx for ONTAP S3 Access Points serverless patterns collection. Building on the Phase 1 foundation and the 14 industry patterns from Phase 2, Phase 3 adds three cross-cutting capabilities:

Near-real-time processing: Kinesis Data Streams integration for minute-level change detection with seconds-level downstream processing after events are emitted (UC11)
ML inference pipeline: SageMaker Batch Transform with Step Functions Callback Pattern for point cloud segmentation (UC9)
Observability stack: X-Ray distributed tracing + CloudWatch EMF metrics across all 14 use cases

Streaming and SageMaker features are opt-in via CloudFormation Conditions (default disabled, zero additional cost). Observability features (X-Ray, EMF, Dashboard, Alarms) can also be disabled but are enabled by default in the reference deployment.

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

Introduction

In Phase 1 and 2, we established a polling-based architecture using EventBridge Scheduler + Step Functions to process files stored on FSx for ONTAP via S3 Access Points. This approach works well for batch workloads with hourly or daily processing cycles, but some use cases demand faster response times.

Phase 3 addresses three gaps:

Latency: Kinesis does not remove the need to detect changes — FSx for ONTAP S3 AP does not provide native event notifications. Instead, it decouples the discovery cadence (minute-level polling) from downstream processing and enables low-latency fan-out once change events are emitted.
ML Integration: Large-scale inference jobs (like LiDAR point cloud segmentation) need asynchronous execution without blocking the Step Functions workflow.
Visibility: As the pattern collection grows to 14 use cases, operators need centralized metrics, distributed tracing, and automated alerting.

Summary Table

Feature	Component	AWS Services	Verification
Near-Real-Time Streaming	Stream Producer + Consumer (UC11)	Kinesis Data Streams, DynamoDB, Lambda	✅ E2E (PutRecord → GetRecords → DynamoDB state transition)
ML Inference	SageMaker Batch Transform (UC9)	SageMaker, Step Functions Callback	✅ Mock mode (Task_Token round-trip verified)
Distributed Tracing	X-Ray instrumentation (all 14 UCs)	AWS X-Ray	✅ X-Ray support added across all Lambda templates (Active by default)
Custom Metrics	EMF output (all 14 UCs)	CloudWatch EMF	✅ 573 tests pass (EMF JSON round-trip property)
Centralized Dashboard	CloudWatch Dashboard	CloudWatch	✅ Deployed (FSxN-S3AP-Patterns-Dashboard)
Alert Automation	CloudWatch Alarms + SNS	CloudWatch, SNS, KMS	✅ Deployed (composite + KMS-encrypted SNS; 15 alarms in reference deployment)
Change Detection	DynamoDB state table	DynamoDB	✅ E2E (pending → completed state transition)

Cost Impact

Feature	Default	Monthly Cost (when enabled)
Kinesis Streaming	Disabled	~$14/month (1 shard, ap-northeast-1; approximate, varies by region and retention settings)
SageMaker Batch Transform	Disabled	Pay-per-job (no persistent endpoint)
X-Ray Tracing	Enabled	Depends on trace volume; free tier includes 100K traces
CloudWatch EMF	Enabled	Included in CloudWatch Logs pricing
Dashboard + Alarms	Enabled	Varies by alarm count; reference deployment: 15 alarms + 1 dashboard

All features can be individually disabled via CloudFormation parameters (EnableStreamingMode, EnableSageMakerTransform, EnableXRayTracing, EnableCloudWatchAlarms). In cost-sensitive PoC environments, disable X-Ray and alarms if they are not needed.

Architecture Decision: Streaming vs Polling

Why Both Patterns?

FSx for ONTAP S3 Access Points don't support GetBucketNotificationConfiguration — there's no native event notification when files change. This means we must actively detect changes. The question is: how frequently, and how do we decouple detection from processing?

Polling (Phase 1/2 approach):

EventBridge Scheduler (rate(1 hour)) → Step Functions → Discovery Lambda → Processing

Streaming (Phase 3 addition):

EventBridge (rate(1 min)) → Stream Producer (detect changes) → Kinesis → Stream Consumer (process) → Pipeline

The key insight: Kinesis doesn't detect changes faster — the Stream Producer still polls at 1-minute intervals. What Kinesis provides is decoupled, low-latency fan-out once change events are emitted. The consumer processes events within seconds of them appearing on the stream.

When to Use Each

Criteria	Polling	Streaming
Detection latency	Configurable (min 1 min)	1 minute (producer polling)
Processing latency after detection	Seconds to minutes (Step Functions)	Seconds (Kinesis consumer)
File change rate	< 1,000/hour	> 1,000/hour
Cost priority	✅ Lower at low volume	✅ Lower at high volume
Operational simplicity	✅ Simpler	More components
Failure handling	Step Functions Retry/Catch	bisect-on-error + DLQ

The Hybrid Approach (Recommended)

For production deployments, we recommend running both:

Streaming handles near-real-time processing (seconds-level latency after detection)
Polling runs hourly as a consistency reconciliation pass

This gives you the best of both worlds: fast downstream processing with guaranteed eventual consistency. If the streaming path fails, the polling path catches up automatically.

Kinesis Integration Deep Dive

Stream Producer Design

The Stream Producer Lambda runs every minute via EventBridge Scheduler. Its job is change detection:

Call ListObjectsV2 on the S3 Access Point to get the current file listing
Compare against a DynamoDB state table (partition key: file_key, attributes: last_modified, etag, processing_status)
For each new or modified file, write a change event to Kinesis Data Streams

# Change detection logic (simplified for illustration)
# Production implementation should avoid full table scans for large namespaces.
# Use paginated scans, BatchGetItem for listed keys, or prefix-partitioned state tracking.
current_objects = s3_client.list_objects_v2(Bucket=s3_ap_alias)
stored_state = dynamodb_table.scan()

for obj in current_objects:
    stored = stored_state.get(obj['Key'])
    if not stored or stored['etag'] != obj['ETag']:
        # New or modified file detected
        streaming_helper.put_records([
            create_event_record(
                key=obj['Key'],
                event_type='CREATED' if not stored else 'MODIFIED',
                timestamp=datetime.utcnow().isoformat(),
                metadata={'size': obj['Size'], 'etag': obj['ETag']}
            )
        ])

Scaling note: For clarity, the snippet uses scan(). In production with large namespaces (10K+ objects), use paginated scans, BatchGetItem for the keys returned by ListObjectsV2, or prefix-partitioned state tracking to avoid full-table scans on every producer run.

Partition Key Strategy

The partition key is derived from the file path prefix (first directory level). This ensures files in the same directory land on the same shard, enabling ordered processing within a directory while distributing load across shards.

Hot shard risk: If most files land under the same prefix (e.g., all in products/), this strategy can create a hot shard. For high-throughput workloads, consider adding a hash suffix or implementing a configurable partitioning strategy.

Stream Consumer and Dead-Letter Handling

The Stream Consumer Lambda is triggered by Kinesis Event Source Mapping with:

Batch size: 10 records
bisect-on-error: Enabled (splits failed batches to isolate problematic records)
Maximum retry attempts: 3

Failed records are captured by the consumer logic and written to a DynamoDB dead-letter table for investigation. This is a custom implementation within the consumer Lambda — not the event source mapping's built-in on-failure destination (which targets SQS/SNS). The DynamoDB DLQ stores:

Original record data
Error message and stack trace
Timestamp of failure
Retry count

This avoids blocking the entire shard while preserving failed records for manual reprocessing.

Idempotent Processing

Since both the streaming and polling paths may process the same file, idempotency is critical. We use DynamoDB conditional writes:

dynamodb_table.update_item(
    Key={'file_key': file_key},
    UpdateExpression='SET processing_status = :status, processed_at = :ts, etag = :etag',
    ConditionExpression='attribute_not_exists(etag) OR etag <> :etag',
    ExpressionAttributeValues={
        ':status': 'COMPLETED',
        ':ts': current_timestamp,
        ':etag': current_etag
    }
)

Note: In production, the condition should include the source object's ETag or last_modified timestamp so that idempotency is tied to the file version, not only to processing time. This prevents stale events (arriving out of order) from overwriting newer processing results.

SageMaker Callback Pattern

The Problem

SageMaker Batch Transform jobs can run for minutes to hours depending on data volume. We can't have a Step Functions state waiting synchronously — that would block the workflow and incur unnecessary costs.

The Solution: .waitForTaskToken

Step Functions' Callback Pattern (.waitForTaskToken) is perfect for this:

The workflow reaches the SageMaker step and pauses, generating a unique Task Token
A Lambda function starts the Batch Transform job, storing a correlation ID
The workflow waits without holding Lambda compute. With Standard Workflows, you pay for state transitions rather than Lambda runtime during the wait.
When the job completes, an EventBridge rule triggers a callback Lambda
The callback Lambda calls SendTaskSuccess or SendTaskFailure with the Task Token

# Step Functions state definition (simplified)
SageMakerTransformStep:
  Type: Task
  Resource: arn:aws:states:::lambda:invoke.waitForTaskToken
  Parameters:
    FunctionName: !Ref SageMakerInvokeLambda
    Payload:
      task_token.$: $$.Task.Token
      input_path.$: $.point_cloud_s3_path
      model_name: !Ref SageMakerModelName

Task Token Propagation

Important: AWS resource tags have value length limits (typically 256 characters). Step Functions Task Tokens can be significantly longer. In the production path, the Task Token should be stored in DynamoDB and correlated with the TransformJobName or a short correlation ID. The SageMaker job tag should store only the correlation ID, avoiding tag value length limits.

Recommended production flow:

Step Functions (.waitForTaskToken)
  → SageMaker Invoke Lambda (receives token in payload)
    → Store TaskToken in DynamoDB keyed by TransformJobName
    → CreateTransformJob (tag: "CorrelationId": "<short-id>")
      → Job completes → EventBridge rule fires
        → SageMaker Callback Lambda
          → Read CorrelationId from job tags / TransformJobName
          → Fetch TaskToken from DynamoDB
          → SendTaskSuccess/SendTaskFailure (returns token to Step Functions)

Mock mode flow (used in this reference implementation):

In mock mode, the Task Token is passed directly since no actual SageMaker job is created and the token doesn't need to survive across service boundaries:

if os.environ.get('MOCK_MODE', 'false') == 'true':
    # Generate mock segmentation output
    mock_labels = [random.randint(0, 10) for _ in range(input_point_count)]
    s3_client.put_object(Bucket=output_bucket, Key=output_key, Body=json.dumps(mock_labels))

    # Directly call SendTaskSuccess (no tag length concern in mock mode)
    sfn_client.send_task_success(
        taskToken=task_token,
        output=json.dumps({'output_path': f's3://{output_bucket}/{output_key}'})
    )

This lets you verify the entire workflow data flow without a trained model or tag length concerns.

Observability Stack

X-Ray Tracing

Every Lambda function and Step Functions state machine in all 14 use cases now supports X-Ray active tracing (enabled by default via EnableXRayTracing=true). This provides:

End-to-end execution visualization: See the complete path from EventBridge trigger through Discovery, Processing, and Report stages
Latency breakdown: Identify which service calls (S3 AP, ONTAP API, Bedrock, Textract) contribute most to execution time
Error correlation: Trace errors back to their source across distributed components

Graceful Degradation

X-Ray SDK is an optional dependency. If not installed or if ENABLE_XRAY is set to false, the tracing decorators become no-ops:

@xray_subsegment(name="s3ap_list_objects", annotations={"use_case": "retail-catalog"})
def list_objects(s3_ap_alias):
    # If X-Ray SDK is unavailable, this decorator does nothing
    return s3_client.list_objects_v2(Bucket=s3_ap_alias)

This means existing deployments continue working without modification — X-Ray is purely additive.

CloudWatch Embedded Metrics Format (EMF)

Every Lambda function emits structured metrics via EMF:

FilesProcessed (Count): Number of files processed per invocation
ProcessingDuration (Milliseconds): End-to-end processing time
ProcessingErrors (Count): Number of errors encountered
BytesProcessed (Bytes): Total data volume processed

EMF writes metrics as structured JSON log lines — no additional PutMetricData API calls needed:

{
  "_aws": {
    "Timestamp": 1700000000000,
    "CloudWatchMetrics": [{
      "Namespace": "FSxN-S3AP-Patterns",
      "Dimensions": [["UseCase", "FunctionName", "Environment"]],
      "Metrics": [
        {"Name": "FilesProcessed", "Unit": "Count"},
        {"Name": "ProcessingDuration", "Unit": "Milliseconds"}
      ]
    }]
  },
  "UseCase": "retail-catalog",
  "FunctionName": "ImageTaggingLambda",
  "Environment": "prod",
  "FilesProcessed": 15,
  "ProcessingDuration": 2340
}

Dashboard and Alerts

A shared CloudFormation template (shared/cfn/observability-dashboard.yaml) creates a CloudWatch dashboard with:

Per-UC widgets: Step Functions success/failure, Lambda error rates, execution duration
Cross-UC aggregation: Total files processed, overall error rate, P50/P90/P99 latency
Kinesis widgets (when streaming enabled): Iterator age, incoming records, consumer lag
SageMaker widgets (when enabled): Job duration, success/failure count

Alert automation (shared/cfn/alert-automation.yaml) provides:

Step Functions failure rate alarms (default: 3 failures in 5 minutes)
Lambda error rate alarms (default: 5% in 5 minutes)
Kinesis iterator age alarms (default: 60 seconds)
Composite alarms for correlated failures
SNS notifications with structured messages (email, optional Slack/PagerDuty)

Multi-Region Deployment

Design Principles

All CloudFormation templates use ${AWS::Region} for dynamic resource construction — no hardcoded region references. This means you can deploy to any region where the required services are available.

Phase 3 Service Availability

Service	Availability	Notes
Kinesis Data Streams	Nearly all commercial regions	Shard pricing varies by region
SageMaker Batch Transform	Nearly all regions	Instance type availability varies
X-Ray	All commercial regions	No constraints
CloudWatch EMF	All commercial regions	No constraints

Pre-Deployment Checklist

Check the AWS Regional Services List
For Kinesis: Verify shard pricing in your target region
For SageMaker: Confirm your desired instance type is available
For Cross-Region UCs (Textract, Comprehend Medical): Confirm target region connectivity

Verification Results

All Phase 3 features were verified in ap-northeast-1 (Tokyo) against a live FSx for ONTAP environment.

AWS Environment Verification

Test	Result	Details
S3 Access Point ListObjectsV2	✅ PASS	Via fsxn-eda-s3ap alias
Kinesis CreateStream + PutRecord + GetRecords	✅ PASS	1 shard, SSE-KMS, partition key routing
DynamoDB State Table CRUD	✅ PASS	PAY_PER_REQUEST, conditional writes
DynamoDB Dead-Letter Table	✅ PASS	record_id partition key
E2E Streaming Pipeline	✅ PASS	Producer → Kinesis → Consumer → DynamoDB
CloudFormation validate-template	✅ PASS	All 14 UC templates
cfn-lint	✅ PASS	0 errors across all templates
CloudWatch Dashboard deploy	✅ PASS	CREATE_COMPLETE
Alert Automation deploy	✅ PASS	CREATE_COMPLETE (KMS + SNS + 3 alarms)
UC11 Full Stack deploy	✅ PASS	36 resources (EnableStreamingMode=true)
UC9 Full Stack deploy	✅ PASS	33 resources (EnableSageMakerTransform=true, MockMode=true)
UC11 Step Functions E2E	✅ SUCCEEDED	Discovery → ImageTagging → CatalogMetadata → QualityCheck (8.974s)
X-Ray Tracing	✅ PASS	TraceId generated, Stream Producer traces visible in X-Ray console
CloudWatch Alarms	✅ PASS	15 alarms active (12 OK, 1 ALARM from duration baseline, 2 INSUFFICIENT_DATA). The ALARM state was expected due to a deliberately low duration baseline (2x multiplier) used for validation.

Local Test Results

Suite	Tests	Status
shared/streaming/	18 (16 unit + 2 property)	✅ All pass
shared/observability.py	23 (19 unit + 4 property)	✅ All pass
retail-catalog (Phase 3)	17 (producer + consumer)	✅ All pass
autonomous-driving (Phase 3)	22 (invoke + callback + properties)	✅ All pass
Total (all UCs)	573	✅ All pass

Property-Based Tests (Hypothesis)

#	Property	Examples	Status
1	StreamingConfig round-trip serialization	100	✅
2	Record batching preserves count and content	100	✅
3	EMF JSON round-trip validity	100	✅
4	xray_subsegment no-op when disabled	100	✅
5	Task_Token propagation round-trip	100	✅
6	Point count invariant (input == output)	100	✅
7	Error state propagation (failed → SendTaskFailure)	100	✅

Lessons Learned

Networking and Access

1. S3 Access Points Don't Appear in Bucket Lists

aws s3api list-buckets and aws s3 ls don't show S3 Access Point aliases. You must access them directly via aws s3 ls s3://<alias>/ or check the FSx console's volume S3 tab. This caught us during initial verification when we thought the access points had been deleted.

2. S3 Access Point IAM Policies Require Two ARN Formats

S3 Access Points require both the alias format (arn:aws:s3:::${alias}) and the ARN format (arn:aws:s3:${region}:${account}:accesspoint/*) in IAM policies. The alias format handles S3 API routing, while the ARN format satisfies IAM policy evaluation. Missing either format results in AccessDenied errors.

At a high level, include both forms in your IAM policy Resource block:

Alias format: arn:aws:s3:::${S3AccessPointAlias} and .../*
Access point ARN format: arn:aws:s3:${Region}:${AccountId}:accesspoint/*

See the Phase 2 article's Design Decisions section or the CloudFormation templates in the repository for the full IAM policy pattern.

3. Verify the Actual DNS and VPC Endpoint Path for S3 Access Points

During verification, S3 AP access from VPC-attached Lambda required careful validation of the DNS resolution path, route table associations, VPC endpoint policies, and access point network origin. Do not assume that creating an S3 Gateway Endpoint alone guarantees successful S3 AP access in every topology.

S3 Access Points can work with both S3 Gateway Endpoints and S3 Interface Endpoints (AWS docs). However, the VPC endpoint policy must allow the required S3 Access Point resources and actions, and the IAM policy must include the ARN formats used by the implementation (see Lesson #2 above). Additionally, if the access point uses VPC origin, ensure enableDnsHostnames and enableDnsSupport are enabled on the VPC.

In our case, the root cause was the S3 Gateway Endpoint not being associated with the Lambda subnet's route table — a simple but easy-to-miss configuration issue.

Verified: After associating the S3 Gateway Endpoint with the correct route table and fixing the IAM policy (two ARN formats), S3 AP access via Gateway Endpoint worked successfully. No S3 Interface Endpoint was needed.

4. Interface VPC Endpoint Security Group Design

Interface VPC Endpoints should use a dedicated security group (separate from the Lambda SG) with an ingress rule allowing HTTPS (443) from the Lambda security group. Using the same SG for both Lambda and Interface VPC Endpoints creates confusion with self-referencing rules and can lead to connectivity issues. Note that Gateway VPC Endpoints do not use security groups — they rely on route table associations and endpoint policies.

Deployment and Packaging

5. DynamoDB Table Creation Timing

DynamoDB tables in PAY_PER_REQUEST mode take 5-10 seconds to become ACTIVE after CreateTable. Immediate PutItem calls will fail with ResourceNotFoundException. In CloudFormation this is handled by dependency ordering, but in scripts always use aws dynamodb wait table-exists.

6. VPC Lambda ENI Cleanup Takes 10-20 Minutes

When deleting CloudFormation stacks with VPC-attached Lambda functions, the ENI (Elastic Network Interface) cleanup can take 10-20 minutes. This is a known AWS behavior. Use --deletion-mode FORCE_DELETE_STACK for stuck DELETE_FAILED stacks.

7. Handler Path Flattening with `aws cloudformation package`

When aws cloudformation package uploads Lambda code to S3, it flattens the directory structure. If your template uses Handler: retail-catalog/functions/discovery/handler.handler, the packaged template must be updated to Handler: handler.handler. We automated this with a sed post-processing step in the deploy script.

8. Lambda Packaging: Individual ZIPs Required for Shared Modules

aws cloudformation package zips the template's directory, but shared modules in parent directories are excluded. For this project, each Lambda function requires an individual ZIP containing both handler.py and the shared/ module at the root level. The deploy script handles this automatically via a package_lambda() helper function.

Workflow and ML Integration

9. Task Token Length and SageMaker Job Tags

AWS resource tags typically have a 256-character value limit. Step Functions Task Tokens can exceed this. For production SageMaker integrations, store the Task Token in DynamoDB and pass only a short correlation ID as a job tag. The mock mode in this reference implementation passes the token directly since no actual SageMaker job is created.

10. Opt-in Design Validates Backward Compatibility

By defaulting streaming and SageMaker features to disabled (CloudFormation Conditions), we confirmed zero impact on existing Phase 1/2 deployments. The same template works for both "Phase 2 mode" (features disabled) and "Phase 3 mode" (features enabled).

Conclusion

Phase 3 transforms the FSx for ONTAP S3 Access Points pattern collection from a batch-oriented toolkit into a near-real-time capable platform with:

Faster downstream processing: Kinesis streaming enables seconds-level processing after minute-level change detection
ML integration: SageMaker Callback Pattern provides scalable, cost-effective inference without persistent endpoints
Production visibility: X-Ray + EMF + Dashboard + Alerts give operators full observability across all 14 use cases

Streaming and SageMaker features are opt-in with zero cost when disabled. Observability is enabled by default but can be individually toggled, maintaining backward compatibility with Phase 1/2 deployments.

What's Next

Event-driven architecture exploration (when FSx ONTAP S3 AP supports native notifications — eliminating the polling requirement entirely)
DynamoDB-based Task Token storage for production SageMaker integrations
Additional ML patterns (real-time inference endpoints, A/B testing)
Multi-account deployment patterns with AWS Organizations

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

This article is part of the "FSx for ONTAP S3 Access Points" series. See Phase 1 and Phase 2 for the foundation.

9 More Industry Serverless Patterns with FSx for ONTAP S3 Access Points — Semiconductor, Genomics, Energy, and Beyond

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Tue, 05 May 2026 14:35:03 +0000

TL;DR

This is Phase 2 of the FSx for ONTAP S3 Access Points serverless patterns collection. Building on the 5 patterns from Phase 1, we add 9 new industry-specific patterns covering semiconductor, genomics, energy, autonomous driving, construction, retail, logistics, education, and insurance.

Key additions:

Cross-region AI/ML: Textract and Comprehend Medical routed from ap-northeast-1 to us-east-1
Large-file / high-object-count building blocks: Streaming download, multipart upload, 10K+ object pagination
Core AI/ML integrations E2E verified via Lambda: Rekognition (15 labels), Textract (text extraction), Comprehend Medical (entity detection), Bedrock (report generation), Athena (SQL queries)
9 CloudFormation stacks deployed: 205 resources, all Step Functions SUCCEEDED

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

Summary Table

UC	Industry	Main Data Types	AWS Services	Verification
UC6	Semiconductor / EDA	GDS, OASIS	Athena, Bedrock	✅ E2E (Athena 4 queries + Bedrock report)
UC7	Genomics	FASTQ, VCF	Athena, Bedrock, Comprehend Medical	✅ E2E (entity detection via cross-region)
UC8	Energy / Oil & Gas	SEG-Y, Well Logs	Athena, Bedrock, Rekognition (optional)	✅ E2E (SEG-Y header + anomaly detection)
UC9	Autonomous Driving	Video, LiDAR	Rekognition, Bedrock	✅ Step Functions SUCCEEDED
UC10	Construction / AEC	IFC, PDF	Textract, Bedrock, Rekognition	✅ Textract cross-region + workflow succeeded
UC11	Retail / E-Commerce	Product Images	Rekognition, Bedrock	✅ E2E (15 labels detected)
UC12	Logistics	Delivery Slips, Images	Textract, Rekognition, Bedrock	✅ E2E (text extraction cross-region)
UC13	Education / Research	PDF Papers	Textract, Comprehend, Bedrock	✅ Step Functions SUCCEEDED
UC14	Insurance / Claims	Photos, Estimates	Rekognition, Textract, Bedrock	✅ E2E (labels + OCR cross-region)

Design Decisions

IAM Policy for S3 Access Points

FSx ONTAP S3 Access Points require two ARN formats in IAM policies. In this implementation, both formats were required to satisfy S3 API access and IAM evaluation paths:

Resource:
  - !Sub "arn:aws:s3:::${S3AccessPointAlias}"        # Alias format (S3 API)
  - !Sub "arn:aws:s3:::${S3AccessPointAlias}/*"
  - !Sub "arn:aws:s3:${AWS::Region}:${AWS::AccountId}:accesspoint/${S3AccessPointName}"  # ARN format (IAM evaluation)
  - !Sub "arn:aws:s3:${AWS::Region}:${AWS::AccountId}:accesspoint/${S3AccessPointName}/*"

VPC Endpoints for Lambda

In the private-subnet / no-NAT deployment model, the Lambda functions need the following endpoints:

Costs are approximate (single-AZ in ap-northeast-1) and vary by region and AZ count.

Endpoint	Type	Cost
Secrets Manager	Interface	~$7.20/mo
FSx	Interface	~$7.20/mo
CloudWatch Monitoring	Interface	~$7.20/mo
CloudWatch Logs	Interface	~$7.20/mo
SNS	Interface	~$7.20/mo
S3	Gateway	Free

Key lesson: The monitoring endpoint is for CloudWatch Metrics, not Logs. You need a separate logs endpoint for Lambda to write CloudWatch Logs from inside a VPC. The SNS endpoint is required for notification publishing from Report Lambda in private subnets.

boto3 Service Name Gotcha

The correct boto3 service name for Comprehend Medical is comprehendmedical (no hyphen), not comprehend-medical. This caused silent failures in early testing where the service was skipped with a WARNING rather than crashing the workflow.

What's New in Phase 2

Cross-Region Client

Textract and Comprehend Medical are unavailable in ap-northeast-1 (Tokyo). Phase 2 introduces a CrossRegionClient that transparently routes API calls to us-east-1:

from shared.cross_region_client import CrossRegionClient, CrossRegionConfig

config = CrossRegionConfig(
    target_region="us-east-1",
    services=["textract", "comprehendmedical"]
)
client = CrossRegionClient(config)

# Textract in us-east-1
result = client.analyze_document(document_bytes=pdf_bytes)

# Comprehend Medical in us-east-1
entities = client.detect_entities_v2(text=medical_text)

The client includes an allow-list to prevent accidental cross-region calls to unintended services, and raises CrossRegionClientError with region and service context for debugging.

Data residency note: For regulated workloads, cross-region invocation should be explicitly reviewed for data residency, audit logging, and compliance requirements. The allow-list in CrossRegionClient is intended to make cross-region behavior explicit rather than implicit.

Streaming Download & Multipart Upload

Phase 2 use cases are designed for large-file and high-object-count workloads such as SEG-Y, FASTQ/VCF, BIM, and media assets. The S3ApHelper now supports:

# Streaming download — never loads entire file into memory
for chunk in s3ap.streaming_download(key="large-file.segy", chunk_size=256*1024):
    process(chunk)

# Range download — read only SEG-Y header (first 3600 bytes)
header = s3ap.streaming_download_range(key="survey.segy", start=0, end=3599)

# Multipart upload — automatic abort on failure
s3ap.multipart_upload(key="output.parquet", data_chunks=chunks, part_size=5*1024*1024)

Discovery Lambda Pagination

For volumes with 10,000+ objects, Discovery Lambda automatically paginates manifests into chunks for Step Functions Map processing.

The 9 New Use Cases

UC6: Semiconductor / EDA — Design File Validation

Detects GDS/OASIS design files, extracts metadata (library name, cell count, bounding box, creation date), aggregates DRC statistics with Athena SQL, and generates design review reports with Bedrock.

Discovery → Map(MetadataExtraction) → DrcAggregation(Athena) → ReportGeneration(Bedrock + SNS)

Services: Athena, Glue Data Catalog, Bedrock (Nova Lite)
Verification: ✅ GDS metadata extracted, Athena 4 queries succeeded, Bedrock report generated

UC7: Genomics / Bioinformatics — Quality Check & Variant Aggregation

Processes FASTQ files for quality metrics (total reads, average quality score, GC content), aggregates VCF variant statistics (SNP count, indel count, Ti/Tv ratio), and generates research summaries with biomedical entity extraction.

Discovery → Parallel[QcMap(FASTQ), VariantMap(VCF)] → AthenaAnalysis → Summary(Bedrock + Comprehend Medical)

Services: Athena, Bedrock, Comprehend Medical (cross-region us-east-1)
Verification: ✅ QC metrics extracted, variants aggregated, Comprehend Medical detected entities from generated biomedical summary text

UC8: Energy / Oil & Gas — Seismic Data Processing

Reads SEG-Y binary headers (first 3600 bytes via range download) for survey metadata, detects anomalies in well log sensor readings using statistical thresholds, and generates compliance reports. Rekognition is used for optional image-based inspection of well-log visualization artifacts.

Discovery → Parallel[SeismicMetadata(Range DL), AnomalyDetection(Well Logs)] → AthenaAnalysis → ComplianceReport(Bedrock + Rekognition)

Services: Athena, Bedrock, Rekognition (well-log image pattern recognition)
Verification: ✅ SEG-Y header parsed, anomaly detection executed, compliance report generated

UC9: Autonomous Driving / ADAS — Labeling Preprocessing

Extracts keyframes from dashcam video, performs Rekognition object detection (vehicles, pedestrians, and other road-scene labels), validates LiDAR point cloud data integrity, and generates COCO-compatible annotation suggestions with Bedrock.

Discovery → Parallel[FrameExtraction(Rekognition), PointCloudQC] → AnnotationManager(Bedrock)

Services: Rekognition, Bedrock
Extension: SageMaker Batch Transform for point cloud segmentation (planned)
Verification: ✅ Step Functions SUCCEEDED

UC10: Construction / AEC — BIM Model Management

Parses IFC files for building metadata, performs version diff detection, OCRs blueprint PDFs with Textract (cross-region), and checks safety compliance rules with Bedrock + Rekognition.

Discovery → Parallel[BimParse(IFC), OcrMap(Textract)] → SafetyCheck(Bedrock + Rekognition)

Services: Textract (cross-region), Bedrock, Rekognition
Verification: ✅ Textract text extraction confirmed, Step Functions workflow succeeded

UC11: Retail / E-Commerce — Product Image Tagging

Detects product images, performs Rekognition label detection with confidence scoring, generates structured catalog metadata with Bedrock, and flags low-quality images for manual review.

Discovery → ImageTagging(Rekognition) → CatalogMetadata(Bedrock) → QualityCheck

Services: Rekognition, Bedrock
Verification: ✅ 15 labels detected (Lighting 98.5%, Light 96.0%, Purple 92.0%)

UC12: Logistics / Supply Chain — Delivery Slip OCR

OCRs delivery slips with Textract (cross-region), normalizes extracted fields with Bedrock, analyzes warehouse inventory images with Rekognition, and generates delivery and routing summary reports.

Discovery → Parallel[OcrMap(Textract), InventoryMap(Rekognition)] → DataStructuring(Bedrock) → Report(Bedrock + SNS)

Services: Textract (cross-region), Rekognition, Bedrock
Verification: ✅ Textract extraction confirmed on generated test PDF, inventory analysis completed

UC13: Education / Research — Paper Classification

OCRs research PDFs with Textract (cross-region), classifies topics with Comprehend, builds citation networks from reference sections, and generates structured metadata.

Discovery → OcrMap(Textract) → Classification(Comprehend + Bedrock) → CitationAnalysis → Metadata

Services: Textract (cross-region), Comprehend, Bedrock
Verification: ✅ Step Functions SUCCEEDED

UC14: Insurance / Claims — Damage Assessment

Detects accident photos and estimate documents, uses Rekognition labels as inputs for preliminary damage triage, OCRs estimates with Textract (cross-region), and generates comprehensive claims reports correlating photo evidence with estimate data.

Discovery → Parallel[DamageAssessment(Rekognition), EstimateOcr(Textract)] → ClaimsReport(Bedrock + SNS)

Services: Rekognition, Textract (cross-region), Bedrock
Verification: ✅ Rekognition labels detected + Textract extracted tracking/estimate text from generated test document

AI/ML Service Verification Results

Core services were verified via Lambda E2E execution (not just direct API calls):

Service	UC	Result
Rekognition DetectLabels	UC11	✅ 15 labels (Lighting 98.5%)
Rekognition DetectLabels	UC14	✅ damage_assessment with labels
Textract DetectDocumentText	UC12	✅ Text extracted from generated test PDF
Textract DetectDocumentText	UC14	✅ Tracking/estimate text extracted from generated test document
Comprehend Medical DetectEntitiesV2	UC7	✅ Entity detection executed on biomedical summary
Bedrock InvokeModel (Nova Lite)	UC6	✅ Design review report generated
Athena StartQueryExecution	UC6	✅ 4 queries (cell_count, bbox, naming, invalid)

Issues Discovered During Phase 2 Verification

#	Issue	Root Cause	Fix
1	Discovery Lambda timeout (300s)	Public subnet + no VPC Endpoints	Private subnet + VPC Endpoints
2	S3 AP AccessDenied	IAM policy missing ARN format	Both Alias + ARN formats
3	Athena RLIKE syntax error	Athena (Trino) doesn't support RLIKE	Use `REGEXP_LIKE()`
4	Missing CloudWatch Logs endpoint	`monitoring` ≠ `logs`	Added separate Logs endpoint
5	Step Functions ItemsPath mismatch	Discovery returns `objects` but SFN expects `fastq_objects`	Added file-type classification
6	Comprehend Medical service name	`comprehend-medical` is invalid	Use `comprehendmedical`
7	Rekognition InvalidImageFormat	284-byte invalid JPEG	Valid 200x200 PNG (56KB)
8	Processing Lambda S3 AP AccessDenied	Only Discovery role had S3 AP permissions	Added to all Processing roles

File-Type Classification in Discovery Lambda

Each UC's Discovery Lambda classifies detected files by type and returns UC-specific keys matching the Step Functions Map ItemsPath:

# UC7 Genomics Discovery returns:
return {
    "objects": all_objects,          # All detected files
    "fastq_objects": fastq_files,   # → QcMap ItemsPath
    "vcf_objects": vcf_files,       # → VariantMap ItemsPath
    "metadata": ontap_metadata,
}

This allows Step Functions to route different file types to different processing branches without additional Lambda invocations.

Deployment

Quick Start (Batch Deploy)

git clone https://github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns.git
cd FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

# Generate deployment templates
./scripts/regenerate_deploy_templates.sh

# Package all Lambda functions
./scripts/deploy_phase2_batch.sh package

# Deploy all 9 stacks
./scripts/deploy_phase2_batch.sh deploy

# Check status
./scripts/deploy_phase2_batch.sh status

Test Data

# Generate and upload test data (GDS, FASTQ, VCF, SEG-Y, IFC, PNG, PDF)
export S3_AP_ALIAS="<your-s3-ap-alias>"
python3 scripts/generate_test_data.py all --upload

Verify shared/ modules

python3 docs/verification-scripts/verify_phase2_shared.py \
  --s3-ap-alias "<your-s3-ap-alias>" \
  --output-bucket "<your-output-bucket>"
# Result: 8/8 PASSED

Cost

Phase 2 uses the same cost-optimized architecture as Phase 1:

Environment	Fixed/mo	Variable/mo	Total/mo
Demo/PoC	~$0	~$1–$3	~$1–$3
Production (1 UC)	~$36	~$1–$3	~$37–$39
Production (all 14 UCs)	~$36	~$14–$42	~$50–$78

VPC Endpoints are shared across all UCs in the same VPC — deploy the first UC with EnableVpcEndpoints=true, subsequent UCs with false. Variable costs depend on object count, document/image size, and AI/ML service usage.

What's Next

SageMaker Batch Transform integration for UC9 (autonomous driving point cloud segmentation)
Real-time streaming with Kinesis for high-frequency sensor data
Multi-account deployment patterns with AWS Organizations
Cost optimization with Lambda Provisioned Concurrency for latency-sensitive UCs

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

Phase 1 Article: Industry-Specific Serverless Automation Patterns with FSx for ONTAP S3 Access Points

Industry-Specific Serverless Automation Patterns with FSx for ONTAP S3 Access Points

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 03 May 2026 20:29:26 +0000

TL;DR

FSx for ONTAP S3 Access Points let you build industry-specific serverless data pipelines against NAS data — without moving files — using EventBridge Scheduler, Step Functions, and AWS AI/ML services. This article introduces 5 use-case patterns and 3 extension patterns, all backed by a reference implementation repository with CloudFormation templates, shared Python modules, and property-based tests.

This is a continuation of FSx for ONTAP S3 Access Points as a Serverless Automation Boundary. While the previous article covered the operational automation layer, this one focuses on concrete, reusable industry patterns with full deployment instructions.

Why Polling-Based? The S3 AP Constraint

S3 Access Points (hereafter S3 AP) expose ONTAP volume data through S3 APIs — ListObjectsV2, GetObject, PutObject, and others. However, GetBucketNotificationConfiguration is not supported, which means S3 event notifications (EventBridge / Lambda triggers) cannot be used.

This is why all patterns in this collection use EventBridge Scheduler + Step Functions for periodic polling:

EventBridge Scheduler (cron/rate)
  └─→ Step Functions State Machine
       ├─→ Discovery Lambda: List objects via S3 AP → Generate Manifest
       ├─→ Map State: Process each object with AI/ML services
       └─→ Report Lambda: Generate results → SNS notification

Architecture: VPC Placement Optimization

A key design decision from verification: only Lambda functions that need ONTAP REST API access are placed inside the VPC. Lambda functions that only use S3 AP (with internet network origin) run outside the VPC.

┌─────────────────────────────────────────────────────┐
│ Inside VPC                                           │
│  - Discovery Lambda (ONTAP REST API + S3 AP)        │
│  - ACL Collection Lambda (ONTAP REST API)           │
│  → Requires VPC Endpoints for Secrets Manager / FSx │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│ Outside VPC                                          │
│  - Processing Lambda (S3 AP + AI/ML services)       │
│  - Report Lambda (S3 + SNS + Bedrock)               │
│  → Direct access to S3 AP (internet origin)         │
│  → No VPC Endpoints needed — cost savings           │
└─────────────────────────────────────────────────────┘

Benefits: Interface VPC Endpoints (~$28.80/month) become unnecessary for most Lambda functions, cold start times improve (no ENI creation), and AI/ML services are accessed directly without NAT Gateway.

Lambda Placement Guide

Purpose	Recommended	Reason
Demo / PoC	Outside VPC	No VPC Endpoints needed, low cost
Production / private network	Inside VPC	Secrets Manager / FSx / SNS via PrivateLink
Athena / Glue use cases	S3 AP network origin: `internet`	AWS managed services need access

Network Origin Constraints

Network Origin	Lambda (outside VPC)	Lambda (inside VPC)	Athena / Glue
internet	✅	✅ (via S3 Gateway EP)	✅
VPC	❌	✅ (S3 Gateway EP required)	❌

Athena and Glue access from AWS managed infrastructure, so they cannot reach VPC-origin S3 APs. Use cases requiring Athena (UC1, UC3) must use internet network origin.

Security and Authorization Model

The solution uses four authorization layers:

Layer	Role
IAM	Controls access to AWS services and S3 Access Points
S3 Access Point	Defines access boundaries through the associated file system user
ONTAP File System	Enforces file-level permissions (UNIX / NTFS ACL)
ONTAP REST API	Exposes only metadata and control-plane operations

Key points:

S3 APIs do not expose file-level ACLs. File permissions are retrieved exclusively via the ONTAP REST API (UC1's ACL Collection uses this pattern)
S3 AP access is authorized on the ONTAP side as the associated UNIX / Windows file system user, after IAM / S3 AP policy checks pass

The 5 Use Cases

UC1: Legal & Compliance — File Server Audit

Collects NTFS ACL information via ONTAP REST API, detects excessive permissions with Athena SQL, and generates natural-language compliance reports with Bedrock.

Services: Athena, Glue Data Catalog, Bedrock | Verification: ✅ E2E success (67/67 Lambda executions)

UC2: Financial Services — Contract & Invoice Processing (IDP)

OCR processing of PDF/TIFF/JPEG documents with Textract, entity extraction with Comprehend, and structured summary generation with Bedrock.

Services: Textract, Comprehend, Bedrock | Verification: ✅ E2E success (Textract via cross-region invocation)

UC3: Manufacturing — IoT Sensor Log & Quality Inspection

CSV sensor logs converted to Parquet for Athena anomaly detection. Inspection images analyzed with Rekognition for defect detection with confidence-based manual review flagging.

Services: Athena, Glue Data Catalog, Rekognition | Verification: ✅ E2E success

UC4: Media — VFX Rendering Pipeline

Detects rendering assets, submits jobs to AWS Deadline Cloud, performs Rekognition quality checks, and writes approved output back to FSx ONTAP via S3 AP PutObject.

Services: Deadline Cloud, Rekognition | Verification: ✅ E2E success

UC5: Healthcare — DICOM Image Classification & Anonymization

Parses DICOM metadata for classification, detects burned-in PII with Rekognition DetectText, and removes PHI with Comprehend Medical.

Services: Rekognition, Comprehend Medical | Verification: ✅ E2E success (Comprehend Medical via cross-region)

Extension Patterns (Verified)

Bedrock Knowledge Bases — RAG

S3 AP as a data source for Bedrock Knowledge Bases. Verified with OpenSearch Serverless + Titan Embed Text v2 (81 documents indexed, Retrieve and RetrieveAndGenerate APIs confirmed).

Transfer Family SFTP — Partner File Exchange

SFTP server connected to S3 AP for external partner file exchange. Verified with SSH public key auth, upload/download operations.

EMR Serverless Spark — Large-Scale Processing

PySpark jobs reading/writing via S3 AP. Verified CSV → Parquet transformation with script and data I/O entirely through S3 AP.

Design Decisions

Shared Modules

All use cases share OntapClient (Secrets Manager auth, urllib3, TLS, retry), FsxHelper (AWS FSx API + CloudWatch metrics), S3ApHelper (pagination, suffix filter), and lambda_error_handler decorator.

Cost Optimization

High-cost always-on resources are opt-in via CloudFormation parameters:

Resource	Monthly Cost	Default
Interface VPC Endpoints (4)	~$28.80	Disabled
CloudWatch Alarms	~$0.10/alarm	Disabled
S3 Gateway VPC Endpoint	Free	Enabled

Demo/PoC cost: ~$1–$3/month. Actual verification cost for all 8 patterns: under $2.

Three-Layer Error Handling

Shared modules: Custom exceptions + urllib3/boto3 retry
Step Functions: Retry/Catch blocks with exponential backoff
Workflow: Map State individual failures don't affect other items

Cross-Region Invocation

Textract and Comprehend Medical are unavailable in some regions (e.g., ap-northeast-1). UC2 and UC5 use TextractRegion and ComprehendMedicalRegion CloudFormation parameters for cross-region API calls.

Note: Cross-region invocation transfers data to another region. Verify data residency and compliance requirements.

Issues Discovered During Verification

#	Issue	Fix
1	`datetime` JSON serialization	Added `default=str`
2	Bedrock Messages API format	Updated to Messages API
3	Athena SQL quoting	Added backtick quoting
4	Lambda package name collision	Added UC prefix to ZIP names
5	S3 Gateway Endpoint duplication	Added `EnableS3GatewayEndpoint` parameter
6	VPC Lambda S3 AP timeout	Added `PrivateRouteTableIds` parameter
7	Textract region unavailability	Added `TextractRegion` cross-region parameter
8	ONTAP self-signed certificate	Added `VERIFY_SSL` environment variable
9	Single route table limitation	Changed to `CommaDelimitedList` type
10	Unnecessary VpcConfig	Removed VpcConfig from S3 AP-only Lambda
11	Comprehend Medical region	Added `ComprehendMedicalRegion` parameter
12	UC4 QualityCheck KeyError	Safe key access pattern
13	pyarrow Lambda layer size	Replaced with stdlib `csv` module

When to Use / When Not to Use

Use this when:

You want to serverlessly process existing NAS data on FSx for ONTAP without moving it
You need file listing and preprocessing from Lambda without NFS/SMB mounts
You want to learn the separation of responsibilities between S3 AP and ONTAP REST API
You want to quickly validate industry-specific AI/ML patterns as a PoC

Don't use this when:

Real-time file change event processing is required (S3 Event Notification not supported)
Full S3 bucket compatibility (Presigned URLs, etc.) is needed
You already have EC2/ECS batch infrastructure with NFS mount operations
File data already exists in standard S3 buckets

Production Readiness Considerations

This repository includes production-oriented design decisions, but actual production environments should additionally consider:

Organizational IAM / SCP / Permission Boundary alignment
S3 AP policy and ONTAP-side user permission review
Audit and execution logs (CloudTrail / CloudWatch Logs)
CloudWatch Alarms / SNS / Incident Management integration
Industry-specific compliance (data classification, PII, PHI)
Data residency for cross-region invocations

Getting Started

git clone https://github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns.git
cd FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

pip install -r requirements.txt
pip install -r requirements-dev.txt
pytest shared/tests/ -v

# Package and deploy (example: UC1)
export AWS_DEFAULT_REGION=us-east-1
./scripts/deploy_uc.sh legal-compliance package
# Then deploy via CloudFormation — see README for full parameter list

The repository includes 8-language READMEs (ja, en, ko, zh-CN, zh-TW, fr, de, es), deployment guides, operations guides, troubleshooting guides, cost analysis, and region compatibility matrix.

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

Yoshiki Fujiwara

FSx for ONTAP S3 Access Points as a Serverless Automation Boundary — AI Data Pipelines, Volume-Level SnapMirror DR, and Capacity Guardrails

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Fri, 01 May 2026 23:44:20 +0000

TL;DR: FSx for ONTAP S3 Access Points let you treat NAS data as an S3-facing automation boundary — without moving data — making serverless AI pipelines and ops workflows practical.

This is a continuation of Building an Agentic Access-Aware RAG System with Amazon FSx for NetApp ONTAP. While the previous article focused on the RAG application itself, this one covers the operational automation layer built around FSx for ONTAP S3 Access Points.

The Shift: S3 Access Points Make Serverless NAS Automation Practical

Enterprise file data lives on FSx for NetApp ONTAP — accessed via SMB/NFS by users and applications every day. Automating operations around that data has traditionally meant mounting NFS from compute instances, managing file system connections, and dealing with the cold-start and connection-limit penalties that come with VPC-mounted Lambda functions.

FSx for ONTAP S3 Access Points change this equation. They expose ONTAP file data through supported S3 object APIs — GetObject, PutObject, ListObjectsV2, and others — while keeping the data in FSx and preserving concurrent SMB/NFS access. The practical shift is that Lambda no longer needs mounted NAS access for the data path. S3 Access Points provide the supported object-facing operations, and ONTAP REST API supplies the storage-system metadata that S3 cannot expose.

This is the architectural pivot that makes the automation suite in this article possible:

Serverless file inventory without NFS/SMB mounts from Lambda
AI / RAG preprocessing directly against ONTAP file data through supported S3 object APIs
Metadata sidecar generation by combining S3-listed objects with ONTAP ACL, export policy, and security-style metadata via REST API
Scheduled governance scans over NAS data using IAM + file-system authorization
Reuse of S3-speaking application components (Bedrock KB, analytics tools) without moving data out of ONTAP

ONTAP REST API complements this by providing the control-plane and storage-system context — volume management, SnapMirror orchestration, capacity monitoring, snapshot operations — that S3 Access Points are not designed to handle.

👉 Code: automation/fsxn-ops/

S3 Access Point Compatibility Model

FSx for ONTAP S3 Access Points support a subset of S3 APIs, not full S3 bucket semantics. Understanding this boundary is essential for building reliable automation.

Supported operations used in this automation:

ListObjectsV2 — file inventory and scanning
GetObject — file content access for preprocessing
PutObject — writing metadata sidecars and manifests

Notable unsupported operations:

GetBucketNotificationConfiguration — this is why the design uses scheduled polling instead of S3 notification-driven triggers
GetObjectAcl / PutObjectAcl — object ACL management is not available
Presign — presigned URL generation is not supported
Bucket-style management APIs and bucket-notification semantics do not apply in the same way here

Authorization operates at two layers: IAM permissions control AWS-level access, while file-system-level authorization uses the mapped user identity (UNIX or Windows) configured on the S3 Access Point. File-level ACLs are retrieved via the ONTAP REST API; S3 APIs do not expose file-system ACLs.

Architecture

The architecture has four layers, with S3 Access Points as the data-path boundary:

┌─────────────────────────────────────────────────────────┐
│  Orchestration: EventBridge Scheduler / Step Functions   │
├─────────────────────────────────────────────────────────┤
│  Compute: Lambda (Python 3.12, VPC-deployed)            │
├──────────────────────┬──────────────────────────────────┤
│  Data Path:          │  Control Plane:                  │
│  FSx ONTAP S3 AP     │  ONTAP REST API                  │
│  (ListObjectsV2,     │  (volumes, SnapMirror,           │
│   GetObject,         │   snapshots, exports,            │
│   PutObject)         │   security_style, ACLs)          │
├──────────────────────┴──────────────────────────────────┤
│  Storage: FSx for NetApp ONTAP (SMB/NFS + S3 AP)       │
└─────────────────────────────────────────────────────────┘

S3 Access Points = data-path (file listing, content access, sidecar writes)
ONTAP REST API = storage metadata + control-plane (resize, SnapMirror, snapshots, ACLs, export policies)
Step Functions / Lambda = orchestration and compute
EventBridge = scheduled execution (polling, not file-change triggers)

Implementation

1. Data Preprocessor: The S3 Access Point Workflow

This is the core use case that S3 Access Points enable. The preprocessor scans FSx ONTAP volumes through S3 Access Points and enriches the results with ONTAP-specific storage metadata.

class DataPreprocessor:
    def list_source_objects(self, prefix="", suffix_filter=None):
        """List files on FSx ONTAP via S3 Access Point (ListObjectsV2)"""
        response = self.s3.list_objects_v2(
            Bucket=self.s3_access_point_arn,  # FSx ONTAP S3 AP ARN
            Prefix=prefix,
        )
        # Filter by extension, collect basic object metadata
        # (key, size, last_modified, etag)
        ...

    def collect_ontap_metadata(self, volume_name):
        """Get storage-system metadata via ONTAP REST API"""
        vol_detail = self.ontap.get_volume(vol_uuid)
        return {
            "security_style": vol_detail["nas"]["security_style"],
            "export_policy": vol_detail["nas"]["export_policy"]["name"],
            "snapshot_count": len(self.ontap.list_snapshots(vol_uuid)),
        }

S3 Access Points give you data-path operations and basic object-facing metadata (key, size, last modified, ETag). ONTAP REST API provides storage-system metadata and control-plane attributes — security style, export policies, snapshot information, and NAS/storage context that S3 APIs cannot expose.

The preprocessor combines both to generate task manifests for downstream AI/analytics pipelines:

S3 AP: ListObjectsV2 → file inventory (.md, .pdf, .docx)
ONTAP REST: GET /storage/volumes → security_style, export_policy, snapshots
  → Generate preprocessing tasks (batch_size=10)
  → Write manifest to S3 (PutObject)
  → Downstream: Bedrock KB Ingestion, analytics, governance

2. ONTAP REST API Client

The shared Python client handles control-plane operations. Credentials come from Secrets Manager.

class OntapClient:
    def __init__(
        self,
        management_lif: str,
        secret_id: str,
        verify_ssl: bool = True,       # Default: TLS verification enabled
        ca_cert_path: str = None,       # CA bundle for production
    ):
        if not verify_ssl:
            logger.warning("TLS verification disabled — lab/PoC only")

TLS verification is enabled by default. FSx for ONTAP management LIFs use self-signed certificates, so production deployments should provide a CA bundle via ca_cert_path. For lab/PoC environments, verify_ssl=False can be set explicitly — the client logs a warning. The ONTAP_VERIFY_SSL and ONTAP_CA_CERT_PATH environment variables control this at the Lambda level.

3. Capacity Monitor with Guardrails

Runs every 5 minutes via EventBridge Scheduler. Checks filesystem-level capacity (FSx API + CloudWatch) and volume-level usage (ONTAP REST API).

Guardrail	Default	Purpose
`DRY_RUN`	`true`	Safe default — logs actions without executing
`MAX_GROW_PER_ACTION_PCT`	50%	Prevents a single run from doubling a volume
`MAX_GROW_PER_DAY_GIB`	500 GiB	Caps total daily expansion
`VOL_THRESHOLD_PCT`	80%	Aligned with AWS recommendation to keep SSD utilization below 80%

Observed behavior: CloudWatch StorageCapacityUtilization metrics are not always available for new filesystems or those with minimal data. The monitor falls back to ONTAP REST API for volume-level monitoring when CloudWatch data is unavailable.

4. Volume-Level SnapMirror Failover Orchestration

Scope: This automation handles planned failover for volume-level SnapMirror relationships — breaking replication, recreating selected CIFS shares and NFS exports on the DR side, and reversing the process for failback. It is not a complete SVM-DR solution. ONTAP's SVM-DR includes additional considerations such as identity preservation and replicated configuration scope. See NetApp's SVM-DR documentation for the full picture.

The Step Functions state machine orchestrates 10 actions through a single Lambda function with action routing.

SnapMirror Initialization: Two Paths

ONTAP supports documented initialization semantics during relationship creation in supported create flows. The transfers API starts an initialize or update operation depending on the current relationship state.

The automation implements both:

initialize action: Creates the relationship with "state": "snapmirrored" in supported create flows. In workflows using a pre-existing destination volume, explicit transfer may be more reliable.
final_transfer action: Explicit POST /snapmirror/relationships/{uuid}/transfers — starts an initialize or update transfer depending on the current state.

Observed in my tested environment (ONTAP 9.17.1P4D3, FSx SINGLE_AZ_1 with pre-existing destination volume): The create-with-state path resulted in a job failure, so the automation fell back to explicit transfer. Both paths are implemented and tested.

Enabling the S3 Access Point Pattern: Private-Subnet Networking

If you want S3 Access Point-driven serverless automation for ONTAP data in a private-subnet design (no NAT Gateway), this is the endpoint footprint you need:

Service	Type	Why
`secretsmanager`	Interface	ONTAP credentials
`fsx`	Interface	FSx API (describe, update)
`monitoring`	Interface	CloudWatch metrics
`sns`	Interface	SNS notifications
`s3`	Gateway	S3 Access Point data-path — must be associated with Lambda subnet's route table (no hourly endpoint charge)

This applies to private-subnet / no-NAT deployments. If your Lambda functions have another egress path, the endpoint requirements differ. The S3 Gateway endpoint specifically needs to be associated with the route table used by the Lambda subnet.

What I Learned from AWS Verification

Deployed and tested against FSx for ONTAP (ONTAP 9.17.1P4D3). The following are empirical observations from my tested deployment pattern.

SNS VPC endpoint is required for alerts — without it, SNS Publish silently times out in private-subnet Lambda. This is documented AWS behavior for VPC-deployed Lambda, but easy to overlook.
fsxadmin password sync is not automatic — Secrets Manager and FSx ONTAP store the password independently. If someone changes it via the console, Lambda gets 401 errors.
S3 Gateway endpoint route table association matters — it must be the specific route table used by the Lambda subnet, not just any route table in the VPC.

Same-SVM SnapMirror on SINGLE_AZ_1: Test Harness Only

⚠️ This is a low-cost automation test harness, not a real DR architecture. Source and destination volumes remain in the same failure domain (same filesystem, same AZ). Use this only for validating automation logic.

I tested SnapMirror within the same SVM on a SINGLE_AZ_1 deployment. It works for validating the automation without the cost of a second filesystem (~$200/month minimum).

Cost

Component	Monthly
Lambda (4 functions)	~$1.65
Step Functions	~$0.05
EventBridge Scheduler	~$0.00
Secrets Manager	~$0.40
CloudWatch Logs	~$0.50
Serverless subtotal	~$2.60
VPC Interface Endpoints (4 × ~$7.30/AZ)	~$29-58
S3 Gateway Endpoint	$0.00
Total (with dedicated endpoints)	~$32-61

If your VPC already has these endpoints, the incremental cost is the serverless subtotal only. The S3 Gateway endpoint has no hourly endpoint charge, so the dominant networking cost comes from the four interface endpoints.

Extending the S3 Access Point Pattern

Automated Permission Metadata Pipeline

The strongest extension connects S3 Access Points to the RAG system's permission pipeline:

EventBridge (daily)
  → S3 AP: ListObjectsV2 → file inventory
  → ONTAP REST: GET ACL metadata per file
  → Generate .metadata.json with allowed_group_sids
  → S3 AP: PutObject → write sidecars alongside source files
  → Trigger Bedrock KB Ingestion Job

This eliminates manual .metadata.json management — the automation reads NTFS ACLs from ONTAP REST API and generates permission metadata automatically, writing it back through the S3 Access Point.

Multi-Volume Ingestion Orchestration

For environments with multiple FSx ONTAP volumes, each with its own S3 Access Point:

Step Functions Map:
  → Per volume: S3 AP scan → ONTAP metadata → generate sidecars
  → Per volume: Trigger Bedrock KB Ingestion Job
  → Wait for all → validate vector counts → notify

ONTAP Operations Chatbot

Combine ontap_api_executor with Bedrock Agent for natural language ONTAP management. The security controls (method restrictions, blocked paths) make this safe for read-only chatbots.

Testing

38 unit tests covering TLS verification modes, SnapMirror dual initialization paths, capacity guardrails, and S3 AP operations:

pip install -r automation/fsxn-ops/requirements.txt
pytest automation/fsxn-ops/tests/ -v

# AWS integration tests (auto-deploys, tests, cleans up)
bash automation/fsxn-ops/tests/integration/run_aws_verification.sh

Wrapping Up

FSx for ONTAP S3 Access Points are the architectural enabler that makes this automation suite practical:

AI / data pipelines — ONTAP file data becomes accessible to serverless workflows through S3 Access Points (supported S3 object APIs), enriched with ONTAP REST API storage-system metadata
Management — ONTAP REST API handles control-plane automation (volumes, snapshots, exports)
Capacity — monitored with guardrails and safe expansion defaults
Volume-level DR — planned failover / failback for volume-level SnapMirror relationships (not full SVM-DR)

The serverless compute cost is ~$2.60/month. The code is open source and deploys with a single CloudFormation command.

👉 GitHub: automation/fsxn-ops/
📖 Full project: Yoshiki0705/FSx-for-ONTAP-Agentic-Access-Aware-RAG

Yoshiki Fujiwara

Building an Agentic Access-Aware RAG System with Amazon FSx for NetApp ONTAP, S3 Vectors, and S3 Access Points— Where AI Respects File Permissions

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 05 Apr 2026 17:15:03 +0000

🆕 Updated April 2026: v4.0.0 released with 6 new features — Agent Registry, Multimodal RAG, Guardrails, Episodic Memory, Voice Chat, and AgentCore Policy. See what's new.

Introduction

Enterprise data lives on file servers. And on those file servers, not everyone can see everything — NTFS ACLs, UNIX permissions, and group policies control who accesses what. But when you plug that data into a Retrieval-Augmented Generation (RAG) system, those permission boundaries tend to disappear. Suddenly, anyone can ask the AI about another team's, division's, or board member's confidential information.

But there's a flip side to this problem that's equally important: without permission awareness, the AI can't fully help the people it should be helping.

Think about it. An engineer has years of design docs, project specs, and team-internal notes in their department's shared folder. A sales lead has pipeline data, customer contracts, and regional forecasts in theirs. When you strip away permissions and dump everything into one vector store, the AI doesn't just leak confidential data — it also drowns each user's results in irrelevant noise from every other team. The engineer gets sales forecasts mixed into their search results. The sales lead gets CI/CD pipeline docs they'll never need.

Permission-aware RAG flips this around. Because the system knows exactly which files each user can access, it delivers personalized, noise-free AI assistance grounded in the data each person actually works with day to day. Your personal folder, your team's shared drive, the cross-functional project space you're part of — the AI sees what you see, nothing more, nothing less.

I built Agentic Access-Aware RAG to make this real. It's an open-source system that lets AI agents autonomously search, analyze, and respond to enterprise data stored on Amazon FSx for NetApp ONTAP — while respecting per-user file-level access permissions. The same question yields different answers depending on who's asking: an admin gets the full financial report, a project member gets their project's restricted docs, and a general user gets public information only. Each user gets an AI assistant that's effectively customized to their role and responsibilities — without any manual configuration.

The entire stack deploys with a single npx cdk deploy --all command.

👉 GitHub: Yoshiki0705/FSx-for-ONTAP-Agentic-Access-Aware-RAG
📦 Latest Release: v4.0.0 — 6 new features added

Architecture at a Glance

Browser → AWS WAF → CloudFront (OAC+Geo) → Lambda Web Adapter (Next.js 15)
                                                    │
              ┌─────────────┬───────────────────────┼──────────────────┐
              ▼             ▼                       ▼                  ▼
        Cognito       Bedrock KB              DynamoDB            DynamoDB
       User Pool    + S3 Vectors /          user-access          perm-cache
                    OpenSearch SL           (SID Data)         (Perm Cache)
                         │
                         ▼
                  FSx for ONTAP
                  (SVM + Volume)
                + S3 Access Point

The system is organized into 7 CDK stacks: WAF, Networking, Security (Cognito), Storage (FSx ONTAP + DynamoDB), AI (Bedrock KB + vector store), WebApp (Lambda + CloudFront), and an optional Embedding stack.

The Core Idea: Permission-Aware RAG

Traditional RAG retrieves documents based on semantic similarity alone. This system adds a second dimension: SID-based permission filtering.

Here's the flow:

User sends a question via the chat UI
The app retrieves the user's SID list (personal SID + group SIDs) from DynamoDB
Bedrock KB Retrieve API performs vector search — each result carries allowed_group_sids metadata
The app matches each document's SIDs against the user's SIDs
Only permitted documents are passed to the Converse API for answer generation
The user sees a filtered response with citation badges showing access levels

■ Admin user: SIDs = [...-512 (Domain Admins), S-1-1-0 (Everyone)]
  public/          → S-1-1-0 match  → ✅ Permitted
  confidential/    → ...-512 match  → ✅ Permitted
  engineering/     → No match       → ❌ Filtered out (no noise from other teams)

■ Engineer (Engineering group member): SIDs = [...-1100 (Engineering), S-1-1-0 (Everyone)]
  public/          → S-1-1-0 match  → ✅ Permitted
  confidential/    → No match       → ❌ Denied
  engineering/     → ...-1100 match → ✅ Their team's docs, front and center

■ Sales user: SIDs = [...-1200 (Sales), S-1-1-0 (Everyone)]
  public/          → S-1-1-0 match  → ✅ Permitted
  confidential/    → No match       → ❌ Denied
  engineering/     → No match       → ❌ No engineering noise in their results

The engineer asking "What's the status of Project X?" gets answers from their team's internal docs — not from sales forecasts or HR policies. The sales lead asking "What are our Q3 targets?" gets their regional data without wading through engineering specs. Each user's AI experience is naturally scoped to the data they work with every day.

S3 Access Points: The Bridge Between FSx ONTAP and Bedrock KB

One of the most impactful recent additions is S3 Access Point integration with FSx for ONTAP. This creates a clean, single-path data ingestion architecture:

FSx ONTAP Volume (/data)
  ├── public/company-overview.md
  ├── public/company-overview.md.metadata.json
  ├── confidential/financial-report.md
  ├── confidential/financial-report.md.metadata.json
      │
      │  S3 Access Point
      ▼
  Bedrock KB Data Source (S3 AP alias)
      │  Ingestion Job (chunking + Titan Embed v2)
      ▼
  Vector Store (S3 Vectors or OpenSearch Serverless)

Before S3 Access Points, getting data from FSx ONTAP into Bedrock KB required either a custom Embedding server with CIFS mounts or manual S3 uploads. Now, Bedrock KB reads documents directly from the FSx ONTAP volume through the S3 Access Point — no intermediate copies, no sync scripts.

The S3 AP user type is automatically selected based on your AD configuration:

AD Configuration	Volume Style	S3 AP User Type	Behavior
AD configured	NTFS	WINDOWS (`Admin`)	NTFS ACLs automatically applied
No AD	NTFS/UNIX	UNIX (`root`)	All files accessible; permission control via `.metadata.json`

One gotcha I discovered: the S3 AP WindowsUser must not include the domain prefix. DEMO\Admin works for CLI operations but causes AccessDenied on data plane APIs (ListObjects, GetObject). Always specify just Admin.

S3 Vectors: Low-Cost Vector Storage

The default vector store is Amazon S3 Vectors — a relatively new service that brings vector search costs down to a few dollars per month, compared to ~$700/month for OpenSearch Serverless.

Configuration	Cost	Latency	Best For
S3 Vectors (default)	~$2-5/month	Sub-second to 100ms	Demo, dev, cost optimization
OpenSearch Serverless	~$700/month	~10ms	High-performance production

S3 Vectors does have a 2KB filterable metadata limit per vector. Since Bedrock KB's internal metadata already consumes ~1KB, custom metadata is effectively limited to ~1KB. The system handles this by setting all metadata keys (including allowed_group_sids) as non-filterable and performing SID matching on the application side after retrieval.

If you start with S3 Vectors and later need higher performance, you can export on-demand to OpenSearch Serverless using the included export-to-opensearch.sh script.

Embedding Design: `.metadata.json` and the Ingestion Pipeline

Permission metadata follows the standard Bedrock KB metadata file specification. Each document has a companion .metadata.json file:

product-catalog.md                    ← Document body
product-catalog.md.metadata.json      ← Permission metadata

The metadata format:

{
  "metadataAttributes": {
    "allowed_group_sids": "[\"S-1-1-0\"]",
    "access_level": "public",
    "doc_type": "catalog"
  }
}

The allowed_group_sids field is a JSON array string of Windows SIDs that are allowed to access the document. S-1-1-0 is the well-known "Everyone" SID.

Bedrock KB Ingestion Jobs automatically read these .metadata.json files alongside documents, chunk the content, vectorize with Amazon Titan Text Embeddings v2 (1024 dimensions), and store everything in the vector store. No custom ETL pipeline needed.

Design Decisions and Trade-offs

At scale (thousands of documents), managing individual .metadata.json files becomes a maintenance burden. The system supports three approaches:

Approach	Status	Pros	Cons
`.metadata.json` (current default)	✅ Production	Bedrock KB native, no extra infra	Doubles file count, manual management
ONTAP REST API auto-generation	✅ Partially implemented	File server ACLs as source of truth	Requires Embedding server
DynamoDB permission master	🔜 Recommended for scale	DB-driven, easy auditing	Requires pre-Ingestion generation pipeline

The recommended direction for large-scale environments:

ONTAP REST API (ACL retrieval)
  → DynamoDB document-permissions table
  → Auto-generate .metadata.json before Ingestion Job
  → Ingest via S3 AP into Bedrock KB

Multiple Authentication Modes

The system supports 5 authentication configurations, all driven by cdk.context.json parameters:

Mode	Authentication	Permission Source	Configuration
A: Email/Password	Cognito native	Manual DynamoDB SID registration	Default (no extra config)
B: SAML AD Federation	Cognito + SAML IdP	AD Sync Lambda → auto SID retrieval	`enableAdFederation=true`
C: OIDC + LDAP	Cognito + OIDC IdP	LDAP query → auto UID/GID retrieval	`oidcProviderConfig` + `ldapConfig`
D: OIDC Claims Only	Cognito + OIDC IdP	OIDC token claims → group mapping	`oidcProviderConfig` + `groupClaimName`
E: SAML + OIDC Hybrid	Both IdPs simultaneously	Combined SID + UID/GID	Both configs + `permissionMappingStrategy=hybrid`

The OIDC/LDAP federation enables zero-touch user provisioning: when a user signs in via the OIDC IdP for the first time, the Identity Sync Lambda automatically queries LDAP for their UID/GID/groups and stores them in DynamoDB. No admin intervention required.

For environments with FSx ONTAP UNIX volumes, the system also supports ONTAP name-mapping — automatically resolving UNIX usernames to Windows users via the ONTAP REST API.

Agentic AI: Beyond Document Search

The system isn't just a search engine. Toggle between three modes with one click:

KB Mode: Permission-aware document search and Q&A
Single Agent Mode: Permission-aware autonomous multi-step reasoning via a single Bedrock Agent
Multi Agent Mode: Supervisor + Collaborator pattern for complex multi-agent workflows

Agent mode includes an Agent Directory — a catalog-style management screen where you can create, edit, share, and schedule Bedrock Agents from templates. The directory now includes a Registry tab for importing agents from AWS Agent Registry, and a Teams tab for creating multi-agent teams.

Permission filtering works in all modes. Even when agents autonomously search and reason across multiple documents, only documents the user is authorized to see are included.

AgentCore Memory (v3.3.0)

With enableAgentCoreMemory=true, the system integrates Amazon Bedrock AgentCore Memory for conversation context maintenance:

Short-term memory: In-session conversation history (TTL: 3 days)
Long-term memory: Cross-session user preferences and summaries (semantic + summary strategies)

Episodic Memory (v4.0.0)

Building on AgentCore Memory, enableEpisodicMemory=true adds a new dimension: the agent remembers how it solved problems, not just what it knows.

While semantic memory stores facts and summaries, episodic memory records complete task episodes — the goal, reasoning steps, actions taken, outcomes, and reflections. When a similar task comes up later, the agent automatically retrieves the top 3 most relevant past episodes and injects them into its reasoning context.

Think of it as giving the agent a "lessons learned" database that grows with every interaction:

Episode recording: After each conversation, a Background Reflection process automatically extracts episodes
Similar episode injection: Before executing a task, the agent searches for similar past episodes and uses them to inform its approach
Episode management UI: Browse, search (semantic, 300ms debounce), and delete episodes from the sidebar
Graceful degradation: If episodic memory fails, core agent functionality continues uninterrupted

The UI shows an "📚 Referenced past experience (N)" badge on responses that leveraged episodic memory.

Additional Features

Smart Routing (v3.1.0)

Automatic model selection based on query complexity. Short factual queries route to Claude Haiku (fast, cheap); complex analytical queries route to Claude Sonnet (powerful). Toggle ON/OFF in the sidebar.

Image Analysis RAG (v3.1.0)

Drag-and-drop image upload in the chat input. Images are analyzed with Bedrock Vision API (Claude Haiku 4.5) and the analysis is integrated into KB search context.

6-Layer Security

Layer	Technology	Purpose
L1	CloudFront Geo Restriction	Geographic access control
L2	AWS WAF (6 rules)	Attack pattern detection
L3	CloudFront OAC (SigV4)	Origin authentication
L4	Lambda Function URL IAM Auth	API-level access control
L5	Cognito JWT / SAML / OIDC	User authentication
L6	SID / UID+GID / OIDC Group Filtering	Document-level authorization

8-Language i18n — Why It Matters

The UI and all documentation (README, guides, setup instructions) are available in 8 languages: Japanese, English, Korean, Simplified Chinese, Traditional Chinese, French, German, and Spanish.

This isn't just a nice-to-have. Enterprise file servers are inherently multi-regional — a global company's FSx ONTAP volumes serve teams across Tokyo, Seoul, Shanghai, Frankfurt, and New York. If the RAG interface only speaks English, you've created a barrier for the very users who need it most.

The implementation uses Next.js next-intl with per-locale message files. Every UI string goes through useTranslations(). The AI's chat responses also match the user's language — a Korean user asking in Korean gets a Korean answer with Korean citation labels.

Here's what the card grid looks like across all 8 languages:

🇯🇵 日本語	🇺🇸 English	🇰🇷 한국어	🇨🇳 简体中文

🇹🇼 繁體中文	🇫🇷 Français	🇩🇪 Deutsch	🇪🇸 Español

v4.0.0: Six New Features (April 2026)

v4.0.0 adds six capabilities that extend the system from document search into a more complete enterprise AI platform. All are opt-in via CDK parameters — zero additional cost when disabled.

Agent Registry Integration

enableAgentRegistry=true adds a "Registry" tab to the Agent Directory, connecting to AWS Agent Registry (Amazon Bedrock AgentCore). Your organization's shared Agents, Tools, and MCP Servers become searchable and importable directly from the UI.

Semantic search across registry records
One-click import from registry to local Bedrock Agent (name collision handling with _imported_YYYYMMDD suffix)
Publish local agents to the registry (with approval workflow)
Resource type filters (Agent / Tool / MCP Server)
Cross-region access via agentRegistryRegion parameter
Fault isolation: registry errors don't affect other Agent Directory tabs

Note: Agent Registry is a Preview API as of April 2026. The implementation uses SigV4-signed HTTP with REST path mapping. When the Node.js SDK adds native commands, the client can be swapped with minimal changes.

Multimodal RAG Search

embeddingModel: "nova-multimodal" switches the Knowledge Base from text-only (Titan Text Embeddings v2) to cross-modal search across text, images, video, and audio using Amazon Nova Multimodal Embeddings.

The architecture uses two patterns that make model changes painless:

Embedding Model Registry: Model definitions are configuration objects in a catalog. Adding a new model = adding one entry
KB Config Strategy: Dynamically generates KB configuration, IAM policies, and Lambda environment variables from the registry entry

For gradual migration, multimodalKbMode: "dual" runs two KBs in parallel — text-only (Titan) + multimodal (Nova) — with a query router that directs text queries to the text KB and image-attached queries to the multimodal KB. Users can toggle between them.

Caveat: Nova Multimodal Embeddings is currently available in us-east-1 and us-west-2 only. Changing the embedding model requires KB recreation and full data re-ingestion.

Guardrails Organizational Safeguards

enableGuardrails=true with optional guardrailsConfig gives fine-grained control over Bedrock Guardrails:

Content filter strength: Per-category (sexual, violence, hate, insults, misconduct, prompt attack) input/output filter levels (NONE/LOW/MEDIUM/HIGH)
Topic policies: Block specific topics (e.g., competitor information)
PII detection: Per-entity-type actions (BLOCK or ANONYMIZE for email, phone, credit card, etc.)
Contextual grounding: Hallucination prevention with configurable thresholds

The UI adds:

GuardrailsStatusBadge on every chat response: ✅ safe / ⚠️ filtered / ⚠️ check unavailable
GuardrailsAdminPanel in the sidebar (admin-only, read-only): shows account guardrails config and detects AWS Organizations Organizational Safeguards
EMF metrics: GuardrailsInputBlocked, GuardrailsOutputFiltered, GuardrailsPassthrough → CloudWatch dashboard + SNS alerts

Error handling follows a Fail-Open strategy: if the Guardrails API times out (5s) or returns 5xx, chat continues normally with an error log. The AI never stops working because of a guardrails hiccup.

Voice Chat (Amazon Nova Sonic)

enableVoiceChat=true adds voice interaction. Click the 🎤 microphone button (or Ctrl+Shift+V), speak your question, and get a text + audio response — all through the same permission-aware RAG pipeline.

Phase 1 (current) uses REST + Bedrock Converse API:

Browser (mic) → POST /api/voice/stream → Converse API (speech→text)
                                        → KB/Agent RAG pipeline
                                        → text + audio response → Browser

Waveform animation (Canvas-based, input=blue, output=green, respects prefers-reduced-motion)
30-second silence timeout with auto-stop
Auto-reconnect (max 3 attempts), then text fallback
Works in KB mode, Single Agent mode, and Multi Agent mode
Permission filtering is input-method-agnostic — voice queries get the same SID/UID/GID filtering as text

Phase 2 (planned) will use API Gateway WebSocket + Nova Sonic InvokeModelWithBidirectionalStream for real-time bidirectional streaming.

Estimated monthly cost: $70–$100 (input ~$0.0019/min, output ~$0.0076/min).

AgentCore Policy

enableAgentPolicy=true adds agent behavior control. Define boundaries in natural language — what tools the agent can use, what APIs it can call, what data it can access — and the system enforces them in real-time.

3 policy templates: Security-focused, Cost-focused, Flexibility-focused
PolicyEvaluationMiddleware: Evaluates every agent action against the policy (3s timeout)
Fail-open / Fail-closed: policyFailureMode controls behavior when policy evaluation fails
Violation logging: EMF-format metrics (PolicyViolationCount, PolicyEvaluationLatency) → CloudWatch dashboard
PolicySection in Agent create/edit forms: optional natural language policy input (max 2000 chars)
PolicyBadge (🛡️) on agents with active policies

Note: AgentCore Policy reached GA in March 2026 with a Policy Engine + Gateway architecture. Policies are written in Cedar language (with natural language auto-conversion). The implementation uses SigV4-signed HTTP.

Feature Flags Runtime API

A cross-cutting change that affects all v4 features: the UI no longer relies on NEXT_PUBLIC_* build-time environment variables. Instead, a /api/config/features endpoint reads Lambda environment variables at runtime and returns feature flags. The useFeatureFlags hook caches flags in localStorage for instant page loads.

This means you can enable/disable features by changing CDK parameters and redeploying — without rebuilding the Docker image.

Multi-Agent Collaboration: Now Default-On

When enableAgent=true, multi-agent collaboration (enableMultiAgent) is now enabled by default. Bedrock Agents have zero standby cost, so this adds no running cost. Token consumption only increases (3-6x) when users actually chat in Multi Agent mode. Set enableMultiAgent: false explicitly to disable.

Multi-Agent Collaboration: Permission-Aware Agent Teams

The system uses Amazon Bedrock Agents' Supervisor + Collaborator pattern. Instead of a single agent handling everything, specialized agents work together:

Supervisor Agent: Detects user intent, routes tasks to the right collaborator
Permission Resolver: Resolves SID/UID/GID from the User Access Table
Retrieval Agent: Executes KB search with permission metadata filters
Analysis Agent: Summarizes and reasons over filtered context (no direct KB access)
Output Agent: Generates reports and documents (no direct KB access)

The key design principle: KB access is restricted to Permission Resolver and Retrieval Agent only. Analysis and Output agents receive "filtered context" — they never touch the knowledge base directly. This preserves the same SID/UID/GID permission boundaries that exist in single-agent mode.

Cost Structure

Scenario	Agent Calls	Est. Cost/Request
Single Agent (existing)	1	~$0.02
Multi-Agent (simple query)	2–3	~$0.06
Multi-Agent (complex query)	4–6	~$0.17

Deployment Lessons Learned

CloudFormation AgentCollaboration values: Only DISABLED, SUPERVISOR, and SUPERVISOR_ROUTER are valid. COLLABORATOR is NOT a valid value. Collaborator Agents should not set this property at all.

2-stage deploy is mandatory: You cannot create a Supervisor Agent with SUPERVISOR_ROUTER and collaborators in a single CloudFormation operation. The solution: create with DISABLED first, then a Custom Resource Lambda changes to SUPERVISOR_ROUTER, associates collaborators, and runs PrepareAgent.

IAM permissions: The Supervisor Agent's IAM role needs bedrock:GetAgentAlias + bedrock:InvokeAgent on agent-alias/*/*. The Custom Resource Lambda needs iam:PassRole for the Supervisor role.

Tips for Builders

OpenLDAP `memberOf` Overlay

If you're testing with OpenLDAP, the LDAP Connector reads the memberOf attribute from user entries. Basic OpenLDAP doesn't populate this automatically — you need to add moduleload memberof and overlay memberof to slapd.conf, and create groupOfNames entries (not just posixGroup).

The repo includes setup-openldap.sh that handles all of this automatically.

Geo Restriction Default

The WAF configuration defaults to Japan-only access (allowedCountries: ["JP"]). If you're deploying outside Japan, update this before deploying:

{ "allowedCountries": ["JP", "US", "DE", "SG"] }

Set to [] for worldwide access.

Existing FSx ONTAP Reuse

If you already have an FSx for ONTAP file system, specify existingFileSystemId, existingSvmId, and existingVolumeId in cdk.context.json to skip FSx creation entirely. This cuts deployment time from 30-40 minutes to under 10 minutes.

Built with Kiro

I used Kiro throughout the entire development lifecycle — specs for requirements-to-code traceability, hooks for automated validation on file saves, and steering files for project-specific rules that persist across sessions. The v4.0.0 release involved 195 files changed, 8-language documentation updates, property-based tests with fast-check, and live AWS environment verification across multiple accounts — all developed with Kiro's assistance. As a solo developer, this level of tooling makes enterprise-quality projects feasible.

Getting Started

git clone https://github.com/Yoshiki0705/FSx-for-ONTAP-Agentic-Access-Aware-RAG.git
cd FSx-for-ONTAP-Agentic-Access-Aware-RAG && npm install

npx cdk bootstrap aws://$(aws sts get-caller-identity --query Account --output text)/ap-northeast-1
npx cdk bootstrap aws://$(aws sts get-caller-identity --query Account --output text)/us-east-1

bash demo-data/scripts/pre-deploy-setup.sh
npx cdk deploy --all --require-approval never
bash demo-data/scripts/post-deploy-setup.sh

Prerequisites: Node.js 22+, Docker, AWS CLI configured with AdministratorAccess. Total deployment time is about 30-40 minutes (FSx ONTAP creation takes 20-30 minutes). Use existingFileSystemId to skip FSx creation if you already have one.

What's Next

The project is at v4.0.0 with 19 implementation aspects and actively evolving. Some directions I'm exploring:

Voice Chat Phase 2: WebSocket via API Gateway + Nova Sonic InvokeModelWithBidirectionalStream for real-time bidirectional streaming (replacing the current REST-based Phase 1)
DynamoDB-driven permission master: Eliminating per-file .metadata.json management for large-scale environments
Multi-volume embedding: Independent S3 Access Points per FSx for ONTAP volume with cross-volume search
Agent Registry GA SDK migration: When the Node.js SDK adds native Agent Registry commands, swap from SigV4 HTTP to SDK calls

I'm looking for feedback on:

Permission models: Are SID/UID-GID/OIDC-group/hybrid strategies sufficient for your use cases?
Voice interaction patterns: What voice-specific workflows would be valuable in enterprise RAG?
Policy templates: What agent behavior boundaries matter most in your organization?
Guardrails configurations: What content filtering rules does your compliance team require?

If you try it out, I'd love to hear about your experience — especially edge cases I haven't considered. PRs and issues are welcome.

👉 GitHub Repository — README available in 8 languages, same as the application UI

Yoshiki Fujiwara

Taking a look at Tiering of AWS ever-evolving File Storage services!

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Tue, 31 Dec 2024 13:42:48 +0000

Disclaimer

Opinions are my own.
Cover image is for Amazon S3 Intelligent-Tiering storage class Automatic Access tiers, but main topic is AWS File Storage Tiering.
If you have any questions or concerns after reading this article, please let us know.
Based on the features and contents as of December 31, 2024. If there are any discrepancies, please check the latest AWS official information at the time you read the article.

Why it's a good time to take a look at AWS File Storage Tiering now?
File storage services on AWS
What is Tiering?
Tierings of AWS file storage services
Conclusion

Why it's a good time to take a look at AWS File Storage Tiering now?

In conclusion, "AWS file storage services and features are so diversified that it is difficult to understand them as a whole". In such a situation, Tiering is an effective function for optimizing the file storage environment. But, it's difficult to understand because there are no materials for cross-sectional understanding.
I would like to understand it from a bird's-eye view rather than comparing services. If I could not find such materials, I'll write it myself.

The direct trigger of this blog post was the update during AWS re:Invent 2024, which introduced the new FSx storage class Amazon FSx Intelligent-Tiering. You can check the AWS release note titled "Announcing Amazon FSx Intelligent-Tiering, a new storage class for FSx"

This is a announcement related storage during AWS re:Invent 2024 and is an exciting update that is attracting attention.

It was also covered at Storage-JAWS#6, the re:Cap community based webinar of AWS re:Invent 2024 of "Storage-JAWS", a storage specialized branch of the Japanese AWS User Group, or JAWS-UG.

You can check the YouTube video linked above for details. (Since this is a local event in Japan, the introduction was in Japanese)

If you haven't seen the Storage-JAWS video above, please check it out as the speakers summarize the updates in an easy-to-understand manner, and there are great sessions and LTs that are explained with live demos and screenshots of management console. Please feel free to fill out the survey after watching (this is a guide for me as a Storage-JAWS management member).

...But,
Contrary to the title and content of the release notes above, "FSx Intelligent-Tiering," the reality is that this is a feature only available for Amazon FSx for OpenZFS, and is not available for the entire Amazon FSx series. Other services that follow are not available at the time of writing this blog.

Personally, I felt that this notation was confusing, so I decided to take this opportunity to clarify "What is File Storage Tiering on AWS and how it works?"

Below are some excerpts from the AWS release note. I think there are some expressions that can lead to misunderstandings as to whether this is about the FSx series or FSx for OpenZFS feature. I will try to write this blog in a way that avoids any misunderstandings.

Today, AWS announces the general availability of Amazon FSx Intelligent-Tiering, a new storage class for Amazon FSx that costs up to 85% less than the FSx SSD storage class and up to 20% less than traditional HDD-based NAS storage on premises, and that brings full elasticity and intelligent tiering to network-attached storage (NAS). The new storage class is available today on Amazon FSx for OpenZFS.

Using Amazon FSx, customers can launch and run fully managed cloud file systems that have familiar NAS capabilities such as point-in-time snapshots, data clones, and user quotas. Before today, customers have been moving NAS data sets for mission-critical and performance-intensive workloads to FSx for OpenZFS, using the existing SSD storage class for predictable high performance. With the new FSx Intelligent-Tiering storage class, customers can now bring to FSx for OpenZFS a broad range of general-purpose data sets, including those with a large proportion of infrequently accessed data stored on low-cost HDD on premises. FSx Intelligent-Tiering delivers low-cost storage and costs up to 85% less than the FSx SSD storage class and up to 20% less than traditional HDD-based NAS storage on premises...

Now, let's get into the details.

File storage services on AWS

Items in this chapter

Storage services on AWS
File Storage services on AWS

Storage services on AWS

First, let's take a look at storage services on AWS.
As far as we can see from the following AWS Webpage "Cloud Storage on AWS", there are 11 "categories" as shown in the figure below, and each category is further divided into services and features.

Quote: Cloud Storage on AWS

File Storage services on AWS

There are three types of file storage in the diagram above: Amazon Elastic File System (EFS), Amazon FSx, and Amazon File Cache. In this article, we will take a look at the EFS and FSx series other than Amazon File Cache, which is a cache service that does not have a tiering feature.
The characteristics of File Storage and how to choose one are summarized in an easy-to-understand manner on the AWS blog "How to choose an AWS file storage service".
It's written in Japanese, but you can easy to understand the contents of it. Or you can check AWS re:Invent sessions of AWS Files Storage Team like AWS re:Invent 2024 - Network-attached storage in the cloud with Amazon FSx (STG202)

For example, in the AWS blog mentioned above, in the "Storage Types" section below, it confirmed the understanding of the three types of "block storage", "object storage", and "file storage" that will be explained this time,

In the "File Storage Protocols" section below, it discusses the two protocols NFS (Network File System) and SMB (Server Message Block), and in the "AWS File Storage Service" section, it explains Amazon FSx for Luster's unique protocol. Let's check the three protocols including.

In addition, in the "Comparison of AWS File Storage Services" section, the figure below shows a list of services excluding FSx for Luster at the time of writing, and points to consider when making a selection. In "Protocols supported by each service," Amazon FSx for NetApp ONTAP is characterized by being multi-protocol, not only supporting both NFS and SMB protocols, but also supporting iSCSI. It also touches on some unique aspects.

Next, in the section "How to choose an AWS file storage service", the part "Consider what you are looking for" touches on Tiering, the main theme of this blog, and the EFS lifecycle policy. In this blog, I will update and supplement the above table based on the AWS re:Invent 2024 Update.

What is Tiering?

What do you think of when you hear the word "Tiering"?
Let's take a look at Amazon S3, AWS's representative storage. In the feature description of "Amazon S3 Intelligent-Tiering storage class" whose name includes Tiering, This feature is introduced from the perspective of cost optimization as follows.

The Amazon S3 Intelligent-Tiering storage class is designed to optimize storage costs by automatically moving data to the most cost-effective access tier when access patterns change.

However, I guess that the scope of implementation and use cases are gradually expanding now.
For example, on an AWS blog, "Optimizing your AWS Infrastructure for Sustainability, Part II: Storage", the section "Analyze data access patterns and use storage tiers", with the following two explanations, it suggests to make S3 lifecycle management sustainable by using automated tiering.

Choosing the right storage tier after analyzing data access patterns gives you more sustainable storage options in the cloud.
For data with unknown or changing access patterns, use Amazon S3 Intelligent-Tiering to monitor access patterns and move objects among tiers automatically.

In this way, we can see that storage tiering is important not only from a cost optimization perspective but also from a sustainability perspective.

I often use not only S3 but also the EFS and FSx series of file storage services from the perspective of optimizing cost and performance, and provide design support. I will take a deep dive into file storage, which has a wide variety of types and features.

Tierings of AWS file storage services

Items in this chapter

Tierings of AWS file storage services
Relevant information

Tierings of AWS file storage services

Description/Service	EFS	FSx for OpenZFS	FSx for ONTAP	FSx for Lustre	FSx for Windows
Tiering	Available	Available	Available	N/A	N/A
Tiering granularity	File Level	Data Block Level	Data Block Level	N/A	N/A
Tiering configuration unit	File System	File System	Volume	N/A	N/A
Tiering Class/Pool	1.Standard 2.Infrequent Access (IA) 3.Archive	1.Frequent Access 2.Infrequent Access 3.Archive	1.Primary Storage(SSD) 2.Capacity Pool (HDD)	N/A	N/A

I purposely included "Tiering granularity". This is because it's one of pitfalls of tiering. Let's say S3 and EFS implement file/object level tiering, so when the file/object is read, it is determined that it has been accessed and the tiering is applied even if the data blocks in the file have hardly been read and tiering won't be triggered the rest of data block has never been accessed.

On the other hand, FSx for OpenZFS and FSx for ONTAP have data block level tiering, so the data blocks that can potentially be optimized for cost/performance through tiering may be wider.

As an example, if you open this blog and leave after just seeing the title, EFS and S3 will not be subject to Tiering who this file/object, but FSx for OpenZFS and FSx for ONTAP will execute Tiering for unread data blocks other than the title. As described in the ONTAP Knowledge Base and Technical Report, FSx for ONTAP judges data blocks in 4K units and performs tiering in 4M units. Regarding FSx for OpenZFS, I have not yet been able to find any documentation that shows the specific behavior of Tiering at the data block level, so if anyone knows about it, I would appreciate it if you could let me know.

Cautions and TIPs:

The FSx Intelligent-Tiering storage class of FSx for OpenZFS can only be used in multi-AZ configurations on a file system basis.
- Screenshot of AWS Management Console for creating FSx for OpenZFS file system. When you choose Intelligent-Tiering (elastic) Storage class, you cannot choose Single-AZ 2 (HA) nor Single-AZ 2 (non-HA).
Tiering in FSx for ONTAP allows you to change and tune the Tiering Policy even after the file system and volume are created. Both single-AZ and multi-AZ configurations are possible.
- Screenshots of AWS Management Console for creating FSx for ONTAP file system and Updating volume.
If you want to run EFS Tiering when creating a file system, select "Customize" and set it in "Lifecycle Management".

Relevant information

EFS Tiering：

Amazon EFS Pricing: > Amazon EFS offers three storage classes: EFS Standard, SSD-based storage which delivers sub-millisecond latencies for actively-used data; EFS Infrequent Access (EFS IA), cost-optimized storage which delivers milliseconds latencies for data accessed only a few times a quarter; and EFS Archive, cost-optimized storage which delivers milliseconds latencies for long-lived data accessed a few times a year or less.
Amazon EFS Infrequent Access：「Amazon EFS will automatically and transparently move your files to the lower cost regional EFS IA storage class based on the last time they were accessed. 」
Amazon EFS Archive：「Amazon EFS will automatically and transparently move your files to the lower cost EFS IA and Archive storage classes based on the last time they were accessed.」

FSx for OpenZFS Tiering：

Amazon FSx for OpenZFS Features > Cost optimization>Intelligent-Tiering(Need to open to see following description):

Intelligent-Tiering delivers automatic storage cost savings when data access patterns change, without performance impact or operational overhead. The Amazon FSx for OpenZFS Intelligent-Tiering storage class is designed to optimize storage costs using elasticity to automatically move data to the most cost-effective access tier when access patterns change. Amazon FSx Intelligent-Tiering is up to 85% lower cost than the FSx SSD storage class, and up to 20% lower cost compared to traditional on-premises HDD deployments.

AWS News blog: Announcing Amazon FSx Intelligent-Tiering, a new storage class for FSx for OpenZFS

FSx for ONTAP：

Amazon FSx for NetApp ONTAP Features > Cost optimization > Elastic capacity pool tiering(Need to open to see following description)：

Each Amazon FSx for NetApp ONTAP file system has two storage tiers: primary storage and capacity pool storage. Primary storage is provisioned, scalable, high-performance SSD storage that’s purpose-built for the active portion of your data set. Capacity pool storage is a fully elastic storage tier that can scale to petabytes in size and is cost-optimized for infrequently-accessed data. Amazon FSx for NetApp ONTAP automatically tiers data from SSD storage to capacity pool storage based on your access patterns, allowing you to achieve SSD levels of performance for your workload while only paying for SSD storage for a small fraction of your data. Capacity pool storage automatically grows and shrinks as you tier data to it, providing elastic storage for the portion of your data set that grows over time without the need to plan or provision capacity for this data.

ONTAP Tiering logic: Technical Report FabricPool best practices ONTAP 9.14.1

Data movement > Tiering data to an object store
After a block has been identified as cold, it is marked for tiering. During this time, a background tiering scan looks for cold blocks. When enough 4KB blocks from the same volume have been collected, they are concatenated into a 4MB object and moved to the cloud tier based on the volume tiering policy.

Conclusion

What do you think of this summary of AWS file storage Tiering?
When selecting a file system, you will most likely choose one that you are familiar with. However, from the perspective of cost/performance optimization, data integration, application configuration, security, etc., why not consider equally the features and benefits of other services?

Among the options discussed this time, there are differences in the tiering methods for EFS, FSx for OpenZFS, and FSx for ONTAP, and you will need to make a choice based on usage and user experience.
Regarding prices, the actual amount and validity vary greatly depending on the method of use and purpose, and it often changes with the release of new features, so I intentionally did not include it in the list this time so as not to make the comparison stand alone.

I am sure that Amazon FSx Intelligent-Tiering will become even more powerful in the future. Let's keep an eye on AWS file storage and Tiering, which will continue to evolve!!
I hope this blog will be helpful to someone.

Bye now!!

Socials:

Forem: Yoshiki Fujiwara(藤原 善基)@AWS Community Builder

Near-Real-Time Processing, ML Inference, and Observability for FSx for ONTAP S3 Access Points — Phase 3 Architecture Patterns

TL;DR

Introduction

Summary Table

Cost Impact

Architecture Decision: Streaming vs Polling

Why Both Patterns?

When to Use Each

The Hybrid Approach (Recommended)

Kinesis Integration Deep Dive

Stream Producer Design

Partition Key Strategy

Stream Consumer and Dead-Letter Handling

Idempotent Processing

SageMaker Callback Pattern

The Problem

The Solution: .waitForTaskToken

Task Token Propagation

Observability Stack

X-Ray Tracing

Graceful Degradation

CloudWatch Embedded Metrics Format (EMF)

Dashboard and Alerts

Multi-Region Deployment

Design Principles

Phase 3 Service Availability

Pre-Deployment Checklist

Verification Results

AWS Environment Verification

Local Test Results

Property-Based Tests (Hypothesis)

Lessons Learned

Networking and Access

1. S3 Access Points Don't Appear in Bucket Lists

2. S3 Access Point IAM Policies Require Two ARN Formats

3. Verify the Actual DNS and VPC Endpoint Path for S3 Access Points

4. Interface VPC Endpoint Security Group Design

Deployment and Packaging

5. DynamoDB Table Creation Timing

6. VPC Lambda ENI Cleanup Takes 10-20 Minutes

7. Handler Path Flattening with aws cloudformation package

8. Lambda Packaging: Individual ZIPs Required for Shared Modules

Workflow and ML Integration

9. Task Token Length and SageMaker Job Tags

10. Opt-in Design Validates Backward Compatibility

Conclusion

What's Next

9 More Industry Serverless Patterns with FSx for ONTAP S3 Access Points — Semiconductor, Genomics, Energy, and Beyond

TL;DR

Summary Table

Design Decisions

IAM Policy for S3 Access Points

VPC Endpoints for Lambda

boto3 Service Name Gotcha

What's New in Phase 2

Cross-Region Client

Streaming Download & Multipart Upload

Discovery Lambda Pagination

The 9 New Use Cases

UC6: Semiconductor / EDA — Design File Validation

UC7: Genomics / Bioinformatics — Quality Check & Variant Aggregation

UC8: Energy / Oil & Gas — Seismic Data Processing

UC9: Autonomous Driving / ADAS — Labeling Preprocessing

UC10: Construction / AEC — BIM Model Management

UC11: Retail / E-Commerce — Product Image Tagging

UC12: Logistics / Supply Chain — Delivery Slip OCR

UC13: Education / Research — Paper Classification

UC14: Insurance / Claims — Damage Assessment

AI/ML Service Verification Results

Issues Discovered During Phase 2 Verification

File-Type Classification in Discovery Lambda

Deployment

Quick Start (Batch Deploy)

Test Data

Verify shared/ modules

Cost

What's Next

Industry-Specific Serverless Automation Patterns with FSx for ONTAP S3 Access Points

TL;DR

Forem: Yoshiki Fujiwara(藤原善基)@AWS Community Builder

7. Handler Path Flattening with `aws cloudformation package`

Embedding Design: `.metadata.json` and the Ingestion Pipeline