Stack Overflow
Stack Overflow

Questions tagged [sharding]

Sharding is a technique of partitioning database tables by row ("horizontally"); typically this technique requires a key to be selected that determines how the rows are to be partitioned.

Sharding is a concept in database design; it refers to the technique of physically partitioning a table or collection by row (also known as horizontal partitioning). To execute the partition, a key or key collection must be defined, which tells the database engine how to determine to which partition each record should belong.

References

1666 questions
11
votes
3 answers

How many shards in a Google App Engine sharded counter?

I read today about sharded counters in Google App Engine. The article says that you should expect to max out at about 5/updates per second per entity in the data store. But it seems to me that this solution doesn't 'scale' unless you have some way…
brainjam
  • 18,863
  • 8
  • 57
  • 82
11
votes
6 answers

How to avoid concurrency issues when scaling writes horizontally?

Assume there is a worker service that receives messages from a queue, reads the product with the specified Id from a document database, applies some manipulation logic based on the message, and finally writes the updated product back to the database…
zafeiris.m
  • 4,339
  • 5
  • 28
  • 41
11
votes
5 answers

Searching across shards?

Short version If I split my users into shards, how do I offer a "user search"? Obviously, I don't want every search to hit every shard. Long version By shard, I mean have multiple databases where each contains a fraction of the total data. For (a…
Eshan
11
votes
2 answers

Database sharding on Heroku

At some point in the next few months our app will be at the size where we need to shard our DB. We are using Heroku for hosting, Node.js/PostgreSQL stack. Conceptually, it makes sense for our app to have each logical shard represent one user and all…
raviparikh
  • 295
  • 1
  • 4
  • 11
11
votes
3 answers

How does MongoDB do both sharding and replication at the same time?

For scaling/failover mongodb uses a “replica set” where there is a primary and one or more secondary servers. Primary is used for writes. Secondaries are used for reads. This is pretty much master slave pattern used in SQL programming. If the…
alex sundukovskiy
  • 483
  • 1
  • 7
  • 19
11
votes
3 answers

NewSQL versus traditional optimization/sharding

We're a small startup with a write-heavy SAAS app and are (finally!) getting to the point where our usage is presenting scaling issues. We have a small team, so we really appreciate being able to offload sysadmin to Heroku and RDS. While Heroku is…
michael
  • 2,977
  • 3
  • 20
  • 26
10
votes
2 answers

Sharded load balancing for stateful services in Kubernetes

I am currently switching from Service Fabric to Kubernetes and was wondering how to do custom and more complex load balancing. So far I already read about Kubernetes offering "Services" which do load balancing for pods hidden behind them, but this…
Sossenbinder
  • 4,852
  • 5
  • 35
  • 78
10
votes
1 answer

Elasticsearch - Assigning Shards

I have recently discovered Elasticsearch and I decided to have a play. Unfortunately I am having trouble with adding indexes. The code used to add an index is as follows and runs every time a new index is attempted to be added: public void index (…
My Head Hurts
  • 37,315
  • 16
  • 75
  • 117
10
votes
3 answers

How to disable Cosmos DB shard key for Mongo API

I'm migrating a very simple mongo DB (couple 100 entries) to Azure Cosmos DB. My app is based on node-js so I'm using mongoose as a mapper. Before it was really simple, define schema, query collection, finished. Now when setting up a collection in…
baouss
  • 1,312
  • 1
  • 22
  • 52
10
votes
3 answers

Elasticsearch on AWS: How to fix unassigned shards?

I have an index on AWS Elasticsearch which were unassighed due to NODE_LEFT. Here's an output of _cat/shards rawindex-2017.07.04 1 p STARTED rawindex-2017.07.04 3 p UNASSIGNED NODE_LEFT rawindex-2017.07.04…
Souradeep
  • 329
  • 1
  • 5
  • 18
10
votes
1 answer

How does PostgreSQL's scaling compare to MongoDB?

I know a fair about about how amazingly scalable MongoDB is, particularly with its auto-sharding, and the ability to hot-add servers to a cluster. However, I haven't found much data about PostgreSQL's scalability. I know it has tablespaces, much…
IamIC
  • 17,747
  • 20
  • 91
  • 154
10
votes
5 answers

Failed to start Sonar Webservice

I am working on Sonarqube 5.1. It used to work fine earlier, later i got this no space left on device error.So i have deleted some unwanted files and restarted the service. Now i can see that the service status is running, but i cannot access the…
sai
  • 300
  • 5
  • 19
10
votes
1 answer

moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because

I have a MongoDb production cluster running 2.6.5 that I recently migrated from two to three shards. I had been running as two shards for about a year. Each shard is a 3-server replica set and I have one collection sharded. The sharded collection…
Jeff Goddard
  • 101
  • 1
  • 3
10
votes
2 answers

Can AUTO_INCREMENT be safely used in a BEFORE TRIGGER in MySQL

Instagram's Postgres method of implementing custom Ids for Sharding is great, but I need the implementation in MySQL. So, I converted the method found at the bottom of this blog, here:…
jsidlosky
  • 407
  • 1
  • 5
  • 17
10
votes
1 answer

Mongodb - sharded and unsharded collections

I'm a bit confused as to how this works. When sharding MySQL, we had some tables, usually small ones with reference data, whole in each shard. This was to enable joins. If we have small collections in MongoDB, that we don't shard in a sharded…
CargoMeister
  • 4,199
  • 6
  • 27
  • 44
1 2 3
…
99 100