Questions tagged [ray]

Ray is a library for writing parallel and distributed Python applications. It scales from your laptop to a large cluster, has a simple yet flexible API, and provides high performance out of the box.

At its core, Ray is a library for writing parallel and distributed Python applications. Its API provides a simple way to take arbitrary Python functions and classes and execute them in the distributed setting.

Learn more about Ray:

Ray also includes a number of powerful libraries:

  • Cluster Autoscaling: Automatically configure, launch, and manage clusters and experiments on AWS or GCP.
  • Hyperparameter Tuning: Automatically run experiments, tune hyperparameters, and visualize results with Ray Tune.
  • Reinforcement Learning: RLlib is a state-of-the-art platform for reinforcement learning research as well as reinforcement learning in practice.
  • Distributed Pandas: Modin provides a faster dataframe library with the same API as Pandas.
702 questions
4
votes
0 answers

Facing issue with integrating code with Aws glue code, ray and pyspark

I am facing the following exception tries various ways but not resolved. It gives the exception in parallel distributed computing processing using ray library Exception: It appears that you are attempting to reference SparkContext from a broadcast…
VISHAL LIMGIRE
  • 529
  • 1
  • 5
  • 21
4
votes
1 answer

Ray[tune] for pytorch TypeError: ray.cloudpickle.dumps

I am having trouble getting started with tune from Ray. I have a PyTorch model to be trained and I am trying to fine-tune using this library. I am very new to Raytune so please bear with me and help me understand where the error stems from. my…
CtrlMj
  • 119
  • 7
4
votes
1 answer

Headless servers Opengym AI rendering Error while using ray

While using ray for distributed computation, all the servers are headless (no display). Therefore, using "xvfb-run -s “-screen 0 1400x900x24” to create screen. Getting error pyglet.canvas.xlib.NoSuchDisplayException: Cannot connect to “None” Without…
4
votes
1 answer

Out of memory at every second trial using Ray Tune

I am tuning the hyperparameters using ray tune. The model is built in the tensorflow library, it occupies a large part of the available GPU memory. I noticed that every second call reports an out of memory error.It looks like the memory is being…
4
votes
0 answers

ray.tune with reuse_actors=True doesn't call reset_config

ray v1.0.0, python 3.7, running ray locally for the below experiment. Following the documentation at: https://docs.ray.io/en/latest/tune/api_docs/trainable.html?highlight=reuse%20actors#advanced-reusing-actors I'm using the Trainable class API and I…
4
votes
2 answers

How to use ray parallelism within a class in python?

I want to use the ray task method rather than the ray actor method to parallelise a method within a class. The reason being the latter seems to need to change how a class is instantiated (as shown here). A toy code example is below, as well as the…
PyRsquared
  • 6,970
  • 11
  • 50
  • 86
4
votes
1 answer

How to explicitly release object after creating it with `ray.put`?

I'm trying to get rid of object pinned in shared memory using ray.put. Here is code sample: import ray for ...: obj_id = ray.put(obj) del obj_id After this is finished,…
ptyshevs
  • 1,602
  • 11
  • 26
4
votes
1 answer

Speeding up PyArrow Parquet to Pandas for dataframe with lots of strings

I have a pandas DataFrame I want to query often (in ray via an API). I'm trying to speed up the loading of it but it takes significant time (3+s) to cast it into pandas. For most of my datasets it's fast but this one is not. My guess is that it's…
Niklas B
  • 1,839
  • 18
  • 36
4
votes
0 answers

Deploy Ray on custom cluster using Docker containers

This question is taken directly from an issue I have opened on the Ray repository, I hope to get more exposure by posting it also here. I saw similar questions both on past issues relative to older versions of Ray and similar problems, but as they…
Luca Cappelletti
  • 2,485
  • 20
  • 35
4
votes
0 answers

How to organize Ray Tune Trainable class calculations in a k-fold CV setup?

It looks like the Ray Tune docs push you to write a Trainable class with a _train method that does incremental training and reports metrics as a dict. There is some persistance of state through the _load _restore methods. TLDR; There is no stated…
mathtick
  • 6,487
  • 13
  • 56
  • 101
4
votes
1 answer

Is there an elegant way to test ray remote functions with pytest?

Say you have a function like this: @ray.remote def remote_function(): return 1 You maybe can test it like: def test_remote_function(): ray.init() x=ray.get(remote_function.remote()) assert x==1 But that means initializing ray…
tteneder
  • 321
  • 2
  • 8
4
votes
0 answers

Is there a way to increase Binary Array capacity in ray/pyarrow?

Is there a way increase BinaryArray limit in pyarrow? I'm hitting this exception when using ray.get: Capacity error: BinaryArray cannot contain more than 2147483646 bytes, have 2147483655
4
votes
1 answer

Can I force task/actor to run on specific Node? using Ray

I'm checking if ray fit to my use-case/business. I know that a group of tasks will connect with the same actor/s and this will cause a lot of IO between the actor/s and the tasks. I want to know if is there a way to force the actor/s and the tasks…
oshribr
  • 658
  • 7
  • 16
4
votes
0 answers

What is the appropriate conv_filter specifications for custom observation_space?

I'm trying to create a custom openai gym environment for use in a ray reinforcement learning environment. Currently I have an observation space of (6, 94). However, when I attempt to train in my custom env, I get the error: ``` ValueError: No…
4
votes
1 answer

How do object_store_memory and redis_max_memory relate?

I don't think I fully understand the relationship between object_store_memory and redis_max_memory. I initially thought that the redis db holds all the objects in memory, but then how can the object store size be made larger than the…
Muppet
  • 5,767
  • 6
  • 29
  • 39
1 2
3
46 47