Requirement
Extract latency, throughput, freshness, payload, quality, explainability, privacy, availability, recovery, and budget constraints.
A decision guide for preparing data, developing models, deploying and orchestrating workloads, monitoring secure production systems, and engineering evaluated generative and agentic AI with responsible LLMOps.
Start with the current AWS exam guide from the certification page. Convert each task statement into three notes: the decision to make, evidence that proves the decision, and the AWS features that may implement it. Then verify behavior in the current service documentation. This protects against memorizing outdated names or assuming a feature behaves identically across Regions, models, endpoint modes, and SDK versions.
Extract latency, throughput, freshness, payload, quality, explainability, privacy, availability, recovery, and budget constraints.
Choose a data, algorithm, inference, orchestration, monitoring, or security pattern and state why alternatives violate a constraint.
Name the metric, test, trace, lineage record, access denial, cost estimate, approval, or rollback exercise that validates the choice.
Reliable ML starts by defining one row at the moment a prediction is made. Record identifiers, event time, ingestion time, label time, source version, and transformation version make leakage and delayed data visible. Fit stateful transforms only on training partitions. For time-dependent tasks, validate with chronological holdouts or rolling backtests rather than random mixing.
| Problem | Preferred response | Evidence | Weak response |
|---|---|---|---|
| Future feature values enter historical rows | Event-time-aware as-of join | No source timestamp exceeds prediction time | Drop the timestamp after joining |
| Training and real-time features differ | Shared versioned definitions; online/offline parity checks | Sampled values match within defined tolerance | Maintain unrelated transformations |
| Rare positive outcome | Task-aligned metrics plus weighted loss or validated resampling | Precision-recall behavior on untouched holdout | Optimize aggregate accuracy |
| Large analytical scans | Columnar formats, useful partitions, bounded file sizes | Measured scan bytes and job time | Create thousands of tiny files |
| Labels arrive late | Stable prediction IDs and a delayed ground-truth join | One outcome joins to the intended prediction | Use drift as a correctness label |
Service pattern: Amazon S3 supplies durable object storage; AWS Glue catalogs metadata and runs data integration; Lake Formation governs data-lake permissions; SageMaker Processing runs managed transformations; SageMaker Feature Store supports governed online and offline features. Select from requirements, not because every design needs every service.
Begin with a simple reproducible baseline. Use a metric connected to business error costs and a holdout that represents deployment. Track the dataset manifest, source revision, container digest, parameters, random seed, environment, metrics, artifacts, and lineage. Automatic tuning is an experiment coordinator, not a substitute for a valid split or an untouched final holdout.
| Signal | Likely diagnosis | Candidate response | Do not infer |
|---|---|---|---|
| Strong training, weak validation | Overfitting, leakage, or distribution mismatch | Audit split and features; regularize; simplify; add representative data; stop earlier | More epochs will fix generalization |
| Both training and validation are weak | Underfit model, poor features, noisy labels, or wrong objective | Inspect labels and features; increase suitable capacity; revisit objective | A larger endpoint improves model quality |
| Model does not fit one device | Model-memory constraint | Model parallel or memory-optimization techniques | Data parallelism always solves fit |
| Training is slow but model fits | Compute, input, or scaling bottleneck | Profile input pipeline; consider data parallelism, instance choice, or optimized formats | Maximum cluster size is automatically economical |
| Probabilities are poorly calibrated | Scores do not correspond well to observed rates | Measure calibration and consider post-hoc calibration on held-out data | High ranking quality guarantees calibrated probabilities |
SageMaker Experiments supports organized run tracking; training jobs provide managed compute; Automatic Model Tuning explores defined hyperparameter ranges; Debugger can capture and analyze tensors and system metrics for supported workloads. Always check current framework, algorithm, and feature support.
| Option | Best starting signal | Key design questions | Operational caution |
|---|---|---|---|
| Real-time endpoint | Synchronous low-latency response | Traffic shape, instance type, auto scaling, availability, p95/p99 | Idle capacity and scaling lag |
| Serverless Inference | Intermittent synchronous traffic | Cold starts, concurrency, model size, memory, supported features | Not every latency target or feature is supported |
| Asynchronous Inference | Longer processing or larger payload pattern; deferred result | Queue, timeout, output destination, notification, scaling | Caller contract must tolerate delayed results |
| Batch Transform | Offline bounded dataset | Partitioning, throughput, reconciliation, completion objective | Not an interactive API |
SageMaker Pipelines is purpose-built for ML workflow steps and lineage integration. Model Registry supplies versioned packages and approval state. Step Functions is useful for broader application orchestration, durable state, and service coordination. EventBridge routes events and schedules work. Whatever the orchestrator, retries around side effects require idempotency, stable execution identifiers, and persisted state.
Safe release sequence: immutable candidate -> offline gates -> registry -> authorized approval -> shadow or canary -> alarms -> gradual traffic -> post-release validation. Retain the known-good version, bind alarms to explicit actions, and exercise both automatic rollback and the kill switch.
| Question | Signal | AWS capability examples | Possible action |
|---|---|---|---|
| Is the service healthy? | Errors, latency, throttles, saturation, availability | CloudWatch metrics, logs, alarms | Scale, fail over, roll back, investigate |
| Did input data change? | Schema, missingness, constraint violations, distribution shifts | Data quality checks, Model Monitor | Block, investigate source, update baseline with review |
| Did prediction quality change? | Metrics after ground truth arrives | Data Capture and Model Monitor model-quality workflows | Investigate, recalibrate, retrain, pause |
| Did responsible behavior change? | Subgroup metrics, bias, explanations, human overrides | SageMaker Clarify plus application metrics and review | Escalate, constrain use, revise data or model |
| Is consumption acceptable? | Instance time, tokens, requests, retries, storage, traces | CloudWatch, Cost Explorer, Budgets, tags | Right-size, cap, cache safely, sample, shorten retention |
Drift is neither an automatic defect nor automatic permission to retrain. A robust response validates data sufficiency, measures actual task quality when labels exist, prevents overlapping jobs, compares against the active baseline, and sends only passing candidates through controlled promotion.
IAM authorization, resource policies, KMS key policies, network paths, service control policies, and application authorization interact. Encryption does not imply authorization, private networking does not imply least privilege, and a content filter does not establish caller identity.
Foundation-model choice should use representative task data and explicit quality, safety, latency, availability, and cost constraints. Public leaderboards can inform a shortlist but cannot prove fitness for a private workflow. Record the exact model identifier and inference settings used in every evaluation.
| Layer | What to evaluate | Typical failure | Remediation direction |
|---|---|---|---|
| Corpus | Authority, freshness, access, format, version, coverage | Stale or restricted document enters retrieval | Govern ingestion and propagate deletion or revocation |
| Chunk and index | Context preservation, metadata, retrieval unit, version | Answer spans are split or lose source identity | Change chunking and preserve metadata |
| Retrieval | Expected-source recall, relevance, authority, filter compliance | Correct evidence is absent or outranked | Improve filters, query, chunking, result count, or reranking |
| Generation | Faithfulness, citations, completeness, abstention, safety | Fluent unsupported claim | Require evidence, validate citations, abstain, or narrow task |
| System | Task success, human outcome, latency, reliability, cost | Good answers are too slow, unsafe, or expensive | Change architecture, model, context, caching, or scope |
Amazon Bedrock provides access to supported foundation models and managed capabilities including Knowledge Bases, Agents, Guardrails, and evaluation features. Availability and feature combinations vary, so verify the current User Guide. Guardrails can add content and safety controls; they are defense in depth, not substitutes for IAM, authorization, schema validation, or human responsibility.
An agent combines nondeterministic reasoning with tools and state. Use it only where flexible planning adds measured value over deterministic code or a simpler RAG workflow. Deterministic components should own authentication, authorization, policy, schema validation, money or record-changing operations, idempotency, approval binding, and terminal-state enforcement.
Narrow verbs, typed inputs and outputs, fixed destinations, least-privilege identity, timeouts, result limits, audit events, and no hidden side effects.
Maximum model calls, tools, loops, retries, tokens, elapsed time, concurrency, and cost with clear terminal reasons.
Show exact action, content, destination, evidence, uncertainty, and consequences; bind approval to that version and revalidate immediately before execution.
Trace model and tool steps, validated arguments or hashes, outcomes, policy decisions, errors, latency, retries, versions, and consumption with content minimized.
LLMOps release unit: model ID and settings + system and task prompts + tools and schemas + policies and guardrails + retrieval corpus and configuration + application code + evaluation dataset and results + infrastructure. Canary and rollback the coupled unit, not just the prompt.
Prompt injection can arrive directly from a user or indirectly through documents, web content, messages, memory, or tool output. Treat that content as untrusted evidence. Do not expose secrets to the model unless strictly necessary, constrain each tool independently, validate every boundary, and require a human for consequential or ambiguous action.
Responsible AI is a lifecycle practice rather than one bias report. Define intended users and excluded uses, affected groups, foreseeable misuse, failure severity, acceptable uncertainty, accessibility needs, escalation, contestability, retention, and human accountability. Compare the AI system with a simpler or no-AI baseline.
| Concern | Question | Evidence | Control examples |
|---|---|---|---|
| Fairness | Who experiences which errors? | Task-relevant disaggregated metrics and impact review | Data review, thresholds, use limits, monitoring, appeal |
| Explainability | Can stakeholders understand important drivers and limits? | Clarify attributions, examples, stability checks, model card | Suitable model, reviewed explanations, human context |
| Privacy | Is each data use necessary and authorized? | Data inventory, purpose, access, retention, deletion proof | Minimization, encryption, isolation, redaction, deletion |
| Safety and robustness | How does the system behave under ambiguity, attacks, and shift? | Adversarial, stress, abstention, rollback, and incident tests | Boundaries, fallback, kill switch, review, monitoring |
| Transparency | Do users know the system role and limitations? | Notices, citations, uncertainty, owner, change records | System card, source display, decision ownership |
Source status was reviewed on 2026-09-11. Recheck the certification page for the current MLA-C02 exam guide and AWS documentation for feature support, quotas, Regions, and prices.