0

In federated/ distributed learning, the server will send initially a global model to clients, and each client will train the model locally and then select the top k values, and send only these values to the server.

How I can select an adaptive k in each client? rather than set top k value to fixed number (e.g. k=3, which return top 3 values), I want to make the top k values adaptive, for example, some clients will send top 4 values, other may send 6 top values based on a defined feature ( largest value, largest loss , ... etc)

Is there any way to do that?

desertnaut
  • 57,590
  • 26
  • 140
  • 166
user18969005
  • 87
  • 6
  • Top k values of what? What is one weight in this case, and how do you select "top" values? – DerekG Aug 29 '22 at 14:28
  • I want to reorder the large weights from clients in descending way, then taking the top large weights. let say some clients may send 3 top weights, other may send top 2 weights. so the top values will be dynamic not static. usually top k values is static (e.g., k = 3 , which takes only top 3 values), how to make it dynamic using a threshold ? is there any algorithm or pseudo code to do that? – user18969005 Aug 30 '22 at 19:46
  • 1
    Well yes, you can easily find the indices of the top k weights (in terms of magnitude) with `torch.topk()` but I'd question whether this approach will learn what you want it to. The largest weights are not necessarily the most crucial to the model's learning and moreover this will create a race condition where weights become larger and larger. You may instead want to find the largest k `grad` values after a backwards pass? – DerekG Aug 30 '22 at 20:25

0 Answers0