IMO we should take it for granted that opium (and perhaps cocaine) has been found to be desirable by at least some of the people who have tried it, resulting in addiction or at least repeated use, and that it is fairly safe or well-tolerated (in that, at the least, users tend not to die of it, at least not immediately).
I think that the main argument against (and the argument in favour of making them illegal) is the cost to society.
Quoting from the Opium article in the Encyclopedia Brittanica of 1911:
The Chinese government regardng the use of opium as one of the most acute moral and economic questions which as a nation they have to face, representing a loss to the country of 856,250,000 taels, decided in 1906 to put an end to the drug within 10 years
and
In October of the same year, the American government in the Philippines, having to deal with the opium trade, ...
and
At this meeting it was resolved that it was the duty of the respective governments to prevent the export of opium to any countries prohibiting its importation
and
The difficulties of the task undertaken by the Chinese government to eradicate a national and popular vice, in a country whose population is generally estimated at 400,000,000, are increased by the fact that opium has been indulged in by all classes of society, that opium has been the principal if not the only national stimulant; that it must involve a considerable loss of revenue, etc.
On page 136 of the same source,
In 1906 is was estimated that 13,455,699 of Chinese smoked opium, or 27% of adult males; but during 1908-1910 the consumption of opium is believed to have diminished by about one third.
The result was the International Opium Convention:
The International Opium Convention, signed at The Hague on January 23, 1912 during the First International Opium Conference, was the first international drug control treaty. It was registered in League of Nations Treaty Series on January 23, 1922.
The text of that convention is easy to find; but I haven't found (except for the Encyclopedia Britannica article mentioned above) a statement of what motivated that agreement. and here is the Report of the International opium commission, Shanghai, China, February 1 to February 26, 1909 (620 pages) which preceded and motivated that agreement. However it seems that commission is more about how to stop the trade: for example the introduction on page 9 states as a given that it's a harmful "bane" (which may be why they called that Commission to discuss it), and the message from the President of the United States is headed "Opium problem", as if the fact that it was problem was common knowledge.
Wikipedia's article about Hampden Coit DuBose says,
As a witness of the destructive problems of opium addiction in China, he was moved to join with other likeminded missionaries and Christian medical workers to form the Anti-Opium League in China. DuBose was its first president. In 1899 the League published a seminal book called Opinions of Over 100 Physicians on the Use of Opium in China. The book illustrated the purpose of the league and published facts about the opium abuse crisis that ultimately influenced public opinion against the trade. DuBose eventually gained the support of U.S. President Theodore Roosevelt, the U.S. Congress, and the International Opium Commission. In 1906 the League achieved success when the British Parliament declared that the trade was "morally indefensible." DuBose circulated a petition signed by over a thousand missionaries in China and it was presented to the Guangxu Emperor. In turn the Emperor issued an imperial edict, following verbatim the petition Dubose had drafted, which prohibited the trade and abuse of opium.
In a letter to U.S. Senator John McLaurin he called upon the U.S. to own its responsibility for the opium trade, in that, along with Great Britain it had profited at the expense of the Chinese. He asserted:
“ …opium has no judicious use…save as administered by a physician[1] ”
His use of the term "judicious" was I guess because of its addictive properties. For example, Wikipedia's Cocaine dependence article says,
Cocaine addiction continues to be the most difficult to manage behind heroin, and according to some scientists, addiction to cocaine may be almost impossible to stop.[23][24]
Assuming that Opinions of over 100 physicians on the use of opium in China (1899) was as influential as stated, here are some extracts from it. The first chapter is about the "moral, physical, and social" effects.
Page v. Introduction:
The slaves of the habit have their faces shrunken and dark. They become old, infirm, and incapacitated before their time, and all finances are exhausted. This condition is pitiable but it is not the worst -- for those who hold office on their part become greedy and grasping, those who are soldiers become nerveless, and the number of depraved population is increased steadily, while the wealth of the country steadily decreases.
Page 2:
This depends largely on a man's means; if he can afford to eat well and plentifully, opium has not the same deleterious effect on the body as when he is badly off for food; he may not lose flesh, but he loses color and powers of resistance. The moral and social effects can never be good, but the degree of badness depends very largely on circumstances -- the temptations a man is led into, either of place or to obtain the drug, the length of time taken and the amount of the drug used each day.