Category: Analytics
Service: Amazon Kinesis Data Streams
Answer:
An Amazon Kinesis Data Streams application consists of several components that work together to process streaming data:
Data Stream: This is the foundational component of an Amazon Kinesis Data Streams application. It is a durable and scalable stream that ingests and stores data in real-time. The data stream is partitioned, allowing for high throughput and parallel processing of data.
Producer: A producer is a source of data that sends data to the Kinesis data stream. Producers can be software applications, sensors, or other devices.
Consumer: A consumer is an application that reads data from the Kinesis data stream. Consumers can process data in real-time or store it for batch processing later.
Shard: A shard is a sequence of data records in a data stream. Each shard can support up to 1 MB of data per second write throughput, and up to 2 MB of data per second read throughput.
Partition key: A partition key is a string value that is associated with each data record sent to the Kinesis data stream. The partition key is used to determine which shard the record will be placed in.
Record: A record is a unit of data sent to the Kinesis data stream. A record consists of a data blob and an optional partition key.
AWS Kinesis Client Library (KCL): KCL is a set of libraries that simplifies the process of consuming and processing data from a Kinesis data stream. The KCL manages the state of the application, including checkpointing the progress of processing data, handling shard failures, and distributing data processing across multiple instances.
Overall, these components work together to provide a scalable, real-time streaming data processing architecture.
Get Cloud Computing Course here
