In our setup maven builds are running on a TeamCity agent and are copied to the TeamCity server repository. Then I use teamcity-s3-plugin to copy them to S3 bucket. Overall size of the artifacts in the build is 3,3GB and it takes about 20 minutes to publish them to the server. We tested the connection between the TeamCity agent and server and it takes about 5 minutes to copy a file of this size, the speed for sptping data is 11,2 MB/s. It takes 3 minutes to copy all artifacts to S3. What might be the reason for the delay? Here is what I see in the agent log file:
[2015-07-15 09:01:25,235] DEBUG - r.artifacts.impl.HttpDiskCache - Http cache put: new entry [https://ci.adsquare.com/httpAuth/repository/download/adsBackendLiveMvn/1353.tcbuildid/jars/weather-service/
target/weather-service-0.0.1-SNAPSHOT.jar]
[2015-07-15 09:01:26,485] DEBUG - r.artifacts.impl.HttpDiskCache - Http disk cache - cached file from [/home/ads/agent/build01/work/864fff464676bc95/weather-service/target/weather-service-0.0.1-SNAPSHO
T.jar] under key [https://ci.adsquare.com/httpAuth/repository/download/adsBackendLiveMvn/1353.tcbuildid/jars/weather-service/target/weather-service-0.0.1-SNAPSHOT.jar]
... messages of this kind for other artifacts
[2015-07-15 09:01:28,390] DEBUG - jetbrains.buildServer.AGENT - Agent ping from server
[2015-07-15 09:01:48,895] DEBUG - jetbrains.buildServer.AGENT - Agent ping from server
...
[2015-07-15 09:22:19,356] DEBUG - jetbrains.buildServer.AGENT - Agent ping from server
[2015-07-15 09:22:39,862] DEBUG - jetbrains.buildServer.AGENT - Agent ping from server
[2015-07-15 09:22:42,083] INFO - jetbrains.buildServer.AGENT - Done publishing artifacts to 'jars', using 'WebPublisher', total files published: 144
[2015-07-15 09:22:42,083] INFO - jetbrains.buildServer.AGENT - Done publishing artifacts to 'jars', using 'ArtifactsCachePublisher', total files published: 144
[2015-07-15 09:22:42,083] INFO - jetbrains.buildServer.AGENT - Done: Publishing files
So what happens in the 20 minutes when the server pings the agent? Is it the time when the artifacts are copied? Why does it take 4 times longer than copying one file? Are there any configuration options to optimize the process?
And on a separate matter - is there a better way to copy files to S3? Is it possible to do it from agent host without publishing them to TeamCity server first?
