0

I'm participating in a Kaggle competition with 4 other people. We all met in a MOOC by edx.org.

Although we can code using the Apache Spark engine, we don't know how to set up a cluster and install the necessary software to run spark on it.

Ideally, we're looking for a free platform that allows us to focus on the programming.

Do you know any platform that is easy to use and, ideally, free? If there isn't one, can you tell us how to set up the necessary infrastructure to participate in the challenge?

Thank you very much in advance.

cchamberlain
  • 17,444
  • 7
  • 59
  • 72
Paca
  • 69
  • 7

1 Answers1

2

It's not that hard to start a standalone cluster on Linux or OS X using the bundled scripts, which could be sufficient if you can work with one node, or each contribute your development computers to a cluster (on the same LAN).

When you need to scale, AWS EMR is pretty simple.

For a little more money, Databricks offers Spark as a managed service. Which means you really don't have to think too much about running the cluster.

Alister Lee
  • 2,236
  • 18
  • 21
  • Since we all live in different locations, running a cluster on the same LAN is not an option. – Paca Aug 05 '15 at 16:31
  • Since we all live in different locations, running a cluster on the same LAN is not an option. The size of the raw data file (.csv) is roughly 80mb. Can I run this locally or will it take forever to perform a task? – Paca Aug 05 '15 at 17:00
  • While you are developing you may not need to use all your data. 80MB is not that much though. Is Spark the best choice? Have you considered R or Python for your first competition? – Alister Lee Aug 06 '15 at 02:01