Console read AWS Kinesis stream

May 1, 2024

What's inside this article ⌄

Read Kinesis stream console
View Kinesis messages AWS
AWS Kinesis stream monitoring
Kinesis data stream viewer

Suppose we want to read AWS Kinesis Data Stream messages from the console. We can make it happen, as long as we have AWS CLI installed and configured.

Get main stream information:

aws kinesis describe-stream-summary --stream-name my-kinesis-stream-name

Get stream shard list:

aws kinesis list-shards --stream-name my-kinesis-stream-name

For intance, we want to read shardId-000000000000. We should get shard iterator for the beginning of the stream for this shard. Remember to insert shard id from the command above:

aws kinesis get-shard-iterator \
  --stream-name my-kinesis-stream-name \
  --shard-id shardId-000000000000 \
  --shard-iterator-type TRIM_HORIZON

Also, we can get shard iterator for the specific point in time (UNIX timestamp):

aws kinesis get-shard-iterator \
    --stream-name my-kinesis-stream-name  \
    --shard-id shardId-000000000000 \
    --shard-iterator-type AT_TIMESTAMP \
    --timestamp 1711582259

Set $iterator equals to iterator from the command above:

$iterator=<123456>

Run command:

aws kinesis get-records --shard-iterator $iterator

Or use this loop:

# Copy NextShardIterator from the output and set $iterator to the new value
# Repeat again.
# Another option, should have jq in order for it to work
# Try before actual usage!
while true; do
  output=$(aws kinesis get-records --shard-iterator $iterator)
  echo "Processing data..."
  iterator=$(echo $output | jq -r '.NextShardIterator')
  [[ -z "$iterator" ]] && break
done

Keep in mind, that you should use awslocal instead of aws, when working with localstack.