I don't want to set a permanent token because it is not secure
Since 13.3 gitlab version there is available Project access tokens, which let you opportunity to issue a token for a specific project and limit it's scope. So it's definitely better than using yourself private token
Project level access tokens allow access to a project without the need to provision a new GitLab user. Project access tokens can be generated by project Maintainers or Owners and be used to authenticate with the GitLab API. Project access tokens will be authorized as Maintainers. This new functionality will make programmatic access to GitLab easier and more secure.
Previous answer:
Probably it would be less insecure if create extra user with permissions only for desired repository (from which we want to download artifacts) and issue personal-token for this user. So even the token will be compromised the offender will be capable only to download artifacts.