7

I wish to create a platform as a service in the financial markets using Erlang/Elixir. I will provide AWS lambda-style functions in financial markets, but rather than being accessible via web/rest/http, I plan to distribute my own ARM-based hardware terminals to clients (Nvidia Jetson TX2-based or similar, so decent hardware). They will access the functions from these terminals. I want said terminals to be full nodes in the system. So they will use the actor model to message pass to my central servers, and indeed, the terminals might message pass amongst each other if terminal users decide to put their own functions online.

Is this a viable model? Could I run 1000 terminals like this? 100 000? What kinds of limitations might I start bumping into? Is Erlang message routing scalable enough to imagine such a network still being performant if we had soft-real time financial markets streaming data flowing around? (mostly from central servers to terminals, but a good proportion possible moving directly around from terminal to terminal). We could have a system where up to 100k or more different "subscription" data channel processes were available, many of them taking input and producing output every second.

Basically I'd like a canonical guide to the scalability capabilities of an Erlang system something like the above. Ideally I'd also like some guide to the security implications of such a system ie. would global routing tables or any other part of the system be compromisable by a rogue terminal user, or can edge nodes be partly "sealed off" from sensitive parts of the rest of the Erlang network?

Note that I'd want to make heavy use of ports/NIFs for high-compute processes.

Thomas Browne
  • 23,824
  • 32
  • 78
  • 121

3 Answers3

7

I would not pursue this avenue for various reasons, all of which hark back to the sort of systems that Erlang's distribution mechanism was developed for - a set of boards on a passive backplane: "free" local bandwidth and the whole machine sits in the same security domain. The Erlang distribution protocol is probably too chatty to work well on widely spread and large networks, and it is certainly too insecure. Unless you want nodes to be able to execute :os.cmd("rm -rf /") on each other, of course.

Use the Erlang distribution protocol in your central system to your heart's content, and have these terminals talk something that's data-only-over-SSL to that system and each other. On top of that, you can quite simply build a sort of overlay network to do whatever you want.

cdegroot
  • 1,765
  • 11
  • 12
4

I recommend read this carefully and i recommend divide your service to little Micro-Services too.
Another benchmark is Investigating the Scalability Limits of Distributed Erlang.
In the Joe Armstorng's book programming Erlang, he said:
"A few years ago, when I had my research hat on, I was working with PlanetLab. I had access to the PlanetLab a network, so I installed empty Erlang servers on all the PlanetLab machines (about 450 of them).
I didn’t really know what I would do with the machines, so I just set up the server infrastructure to do something later."
Do not use External ports, use internal drivers which are written in C or C++ instead.

Pouriya
  • 1,626
  • 11
  • 19
  • okay I assume you mean that each node has n-1 connections meaning n*(n-1) for the whole system? Looks like this fact makes my proposed architecture infeasible because it's essentially a mesh. There isn't actually any routing? Yes I will ask the Erlang mailing list. – Thomas Browne Apr 03 '17 at 08:23
  • 1
    Yes, n-1 is right. I searched for some benchmark and in my last edit, i added them to the post. – Pouriya Apr 03 '17 at 09:26
  • Why wouldn't you use external ports? Also, @Pouriya - please don't answer by giving external references. – cdegroot Apr 03 '17 at 11:49
  • Because i can't perform asynchronous I/O. – Pouriya Apr 03 '17 at 12:34
1

You will find a lot of information regarding erlang Architectures is this answer: How scalable is distributed Erlang?

Short answer is, there is a pratical limitation of nodes in a cluster, but this limitation can be breach with federations fairly easily.

EDIT 1/ Further more I would recommend to read this book : Designing for scalability with Erlang/OTP

Community
  • 1
  • 1
TheSquad
  • 7,385
  • 8
  • 40
  • 79