What will telemetry and monitoring tools show if I lunch in (2 options)
Will they show cpu\memory and etc usage of container only or of host system?
What are best practise? Monitoring software in each container or in host system?
What you want to do is monitor both, the host(s) and the containers running on them. A good way to do that is run a container that collects all data on each docker host. That is how Sematext Docker Agent runs, for example -- it runs as a tiny container on each Docker host and collects all host+containers metrics, events, and logs. It then parses logs, can route them, blacklist/whitelist them, auto-discovery new containers, and so on. In the end logs end up in Logsene and metrics and events end up in SPM, which gives you a single pane of glass sort of view into all your Docker ops bits, with alerting, anomaly detection, correlation, and so on. I hope this helps and points you in the right direction.
The results should be exactly the same, because Docker containers are sharing their resources (unlike virtual machines).
Putting an agent in your containers is not advisable, not just for performance reasons, but it is an anti-pattern in the Docker world, where each container should run a single process. Better is to run a monitoring agent on the host or in a separate container that can be configured to extract metrics from the other containers. This is the way we work at CoScale. If you are interested, have a look at our solution for monitoring Docker.
