Stack Overflow
Stack Overflow

Questions tagged [starcluster]

An open source cluster-computing toolkit for Amazon’s Elastic Compute Cloud (EC2).

StarCluster is an open source cluster-computing toolkit mainly developed in for Amazon’s Elastic Compute Cloud () released under the LGPL license.

StarCluster has been designed to automate and simplify the process of building, configuring, and managing clusters of virtual machines on Amazon’s EC2 cloud. StarCluster allows anyone to easily create a cluster computing environment in the cloud suited for distributed and parallel computing applications and systems.

52 questions
2
votes
1 answer

MIT's Starcluster Errors with too 200+ node

Has anybody had issues bringing up cluster sizes larger than 200 nodes? Whenever I try I get the following error: 7/dist-packages/boto/ec2/connection.py", line 585, in get_all_instances  max_results=max_results)  File…
buckleupbuckaroo
  • 53
  • 7
2
votes
1 answer

Sun Grid Engine, force one job per node

I am running many repeats of the same job using numpy on a cluster that uses sun grid engine to distribute jobs (starcluster). Each of my nodes has 2 cores (c3.large on AWS). So say I have 5 nodes, each with 2 cores. The matrix operations in numpy…
bill_e
  • 930
  • 2
  • 12
  • 24
2
votes
1 answer

Import code from local to all remote engines

I am working on parallelizing an sklearn grid search, sweeping three parameters, but I am having trouble refactoring the project to work with ipython.parallel. My current thought approach has been to create a simple function which: Accepts a a…
Cory Dolphin
  • 2,650
  • 1
  • 20
  • 30
1
vote
0 answers

ipyparallel - 'CannedFunction' object is not callable

I am trying to set up a cluster using StarCluster and ipyparallel. When I try to run the following commands, I get the error below. I have never seen a python error missing a stack trace this way. from ipyparallel import Client rc = Client() ipview…
saq7
  • 1,528
  • 1
  • 12
  • 25
1
vote
1 answer

Distributed computing instance usage with starcluster Ipython parallel plugin

I am using starcluster with Ipython plugin . When I run a Kmeans clustering from Ipython notebook with load balance mode. Its always the Master with 100% CPU usage constantly. And the other EC2 instances never takes the load. I tried with large…
Hanuma Tej
  • 17
  • 4
1
vote
0 answers

Starcluster, how to do a specific task on separate node

Working on a python application that process high statistical data and perform long complex calculation. When user requests an async thread has been created and starts the calculation and save the results in the database at an equal interval, where…
riksof-zeeshan
  • 531
  • 9
  • 27
1
vote
1 answer

AWS Spot Instances and ipcluster plugin

Currently what does the ipcluster plugin do when AWS shuts down one or more of the spot instance nodes? Is there any mechanism to re-start and then re-add these nodes back to the IPython cluster?
Alex Rothberg
  • 10,243
  • 13
  • 60
  • 120
1
vote
1 answer

MPI on an AWS cluster

I've been following the tutorial I found at mpitutorial.com, and I've been able to get by on that so far. However, after setting up my AWS cluster, I have hit a small snag. The program is only running on one processor, even with a host file, and the…
Inglonias
  • 468
  • 1
  • 5
  • 18
1
vote
2 answers

How can I tell if an EC2 instance is inside my VPC?

My client has many EC2 instances running, and a VPC (virtual private cloud) running. I'm using a platform called Starcluster to launch nodes, and I need to know if they're in the VPC or just ordinary EC2 nodes. How can I do that? Amazon's VPC…
rajat banerjee
  • 1,256
  • 2
  • 12
  • 21
1
vote
2 answers

How to dynamically scale StarCluster/qsub/EC2 to run parallel jobs across multiple nodes

I am new to using Startcluster/qsub/grid engine to run parallel jobs and I tried reading couple of other posts regarding the same. I still am not sure of how to build a scalable solution for my specific requirement. I would like to take in some more…
user1652054
  • 445
  • 2
  • 11
  • 23
1
vote
2 answers

The right way to architect a cluster in EC2

I'm working on open-source tool which will have to run on a cluster in EC2, organized in "one master - several slaves" manner. I need some advice on how to organize things correctly and in the most simple, yet reliable way. What I basically need…
lithuak
  • 6,028
  • 9
  • 42
  • 54
1
vote
1 answer

How do you get a StarCluster AMI to work with IPython 0.13?

I tried upgrading IPython on two different StarCluster AMIs (their default 64-bit Ubuntu 11.10 and the QIIME 1.5 image). In either case, when I start up my cluster the initialization script hangs at 'Waiting for JSON connector file...'. I logged in…
Alex Rubinsteyn
  • 420
  • 4
  • 8
1
vote
1 answer

IPython.parallel namespaces

I want to parallelize a function using IPython.parallel, and when I define it in the IPython shell it works flawlessly: Type: function Base Class: String Form: Namespace: Interactive File: …
kyrre
  • 626
  • 2
  • 9
  • 24
0
votes
1 answer

Where is StarCluster Installed on MacOS

I installed "StarCluster" using the terminal on my MacOs by following instructions from the link provided below. Now, I need to edit the configuration file to add my AWS credentials. However, I am not sure which folder "StarCluster" is installed on…
stackuser123
  • 11
  • 5
0
votes
1 answer

StarCluster VolumeCreator AWS instance_type

When attempting to create an EBS shared volume for all nodes in a StarCluster, the StarCluster createvolume command launches an AWS EC2 instance as the volume creator. However, the default instance type that is used for this is t1.micro, which is…
ECHOOooo... ECHOOooo...
  • 11
  • 3
1
2
3 4