-2

I work for a company that is trying to scale some of their workloads by leveraging either Kinesis or managed Kafka (Amazon MSK) as a message queue and streaming service.

All of our systems are running on AWS. Most of our compute workload is handled by Lambda functions, sometimes those Lambdas are behind API Gateway. Probably a significant amount of message production and consumption would happen within Lambda functions, at least at first.

Initially, our use case will be to use the service as a message queue, and we plan to service between 10k and 20k small (100-300 bytes) messages every day.

Eventually, we’ll start using the service in more places by producing and consuming more messages (which could 5-10x the number of messages) and expand our usage of the service into streaming messages to Redshift, S3 and observability dashboards.

I would imagine that for all of these use cases that both services would be more than sufficient.

From the comparison research that I’ve done (which is tricky because many of the resources comparing the two are now out-of-date), it seems like Kinesis is easier to setup and manage, and that for smaller workloads it may be significantly cheaper than MSK.

Given that MSK might be a lot more expensive and require quite a bit more learning, setup, configuration and management, Kinesis seems like an obvious choice.

However, I want to make sure that I’m not missing any important information. Are there any serious limitations or drawbacks to Kinesis that I haven’t considered? Are there important features or capabilities that MSK has that Kinesis doesn’t? Might we regret our choice to use Kinesis if our needs for scalability or flexibility reach a certain point?

I really want to make as responsible of a technology choice as possible for my company.

I’ve tried doing some research but it’s difficult because many of the resources comparing the two services are now out-of-date.

John Smith
  • 45
  • 7
  • 1
    [What types of questions should I avoid asking?](https://stackoverflow.com/help/dont-ask) – Phil Jun 14 '23 at 02:48

1 Answers1

0

This is not a programming question and might be taken down. However, I'd try to answer with my perspective.

Both Kinesis and MSK are production-ready and capable event streaming platforms, and for your use case, it shouldn't matter what you use from an availability and performance point of view.

Two things I would consider:

  1. Cost: Choose the one that provides the best cost-to-value proposition.
  2. Portability to a different Hyperscaler if there is future need. Kinesis an AWS only platform and porting would require complete rewrite. MSK is Kafka behind the scenes and you should be able to port quite easily.
Shameel
  • 632
  • 5
  • 12
  • Where should I post AWS questions that don’t pertain to programming? – John Smith Jun 14 '23 at 03:03
  • 1
    @JohnSmith What I meant to say was that this is not an actual technical question, but rather "which one is better" kind of a question which the site doesn't seem to encourage. – Shameel Jun 14 '23 at 03:06
  • Thanks Shameel. I’m really not sure where else I could take a question like this. I know there’s probably some subjectivity to it, but I tried to keep it as objective as I could. The question is more about comparing the two services objectively so that I can make an informed decision. – John Smith Jun 14 '23 at 03:14
  • 1
    @JohnSmith There's a separate software recommendations StackExchange. Otherwise, Reddit is an option, or AWS support – OneCricketeer Jun 14 '23 at 13:50