Yes. By default, Silverlight is a sandboxed runtime environment. By default, it won't allow you to delete files on the computer where the application is running. However, you can allow it to do this by switching it to Elevated Trust mode. There are basically two ways of doing this. In browser, and out of browser (OOB). Out of browser is simpler, but the user will be required to install the application. Either way, you will need to buy a code signing certificate and sign the Xap package.
For in browser apps, the client's computer will need to be configured explicitly, at the registry level for elevated trust in browser. If the client installs the application as an OOB app, they will not need to do this.
There are many articles on setting up elevated trust, but here is one:
Enabling In-browser elevated trust
Note: I'm a little confused by your use of the word "Server" here. Silverlight, generally speaking is a client side technology. You will not be able to delete files on the server directly. If you need to delete files on the server side, you will need to be build some kind of services on the server side (e.g. WCF, or REST), and call the services from Silverlight on the client side. This may be easier because if you really need to delete files on the server, you will only be able to run the Sivlerlight app on the server, which doesn't seem right.