6

I know that in order to insert values larger than max_allowed_packet bytes into a MySQL database, the default solution would be to assure that both client and server side max_allowed_packet variables are bigger than the chunk of data that a query inserts into the DB.

However, is there any way to do so without changing the server side variable mentioned above? This would be useful when I have to insert data into a database that is hosted in an ISP that doesn't allow me to change the max_allowed_packet limit.

Another related question: MySql longblob limit is 4GB, but max_allowed_packet limit is 1GB. So, is it possible to insert values larger than 1GB in a longblob table column?

Leif Gruenwoldt
  • 13,561
  • 5
  • 60
  • 64
Leandro Gomide
  • 998
  • 1
  • 10
  • 31

2 Answers2

14

I recently stumbled upon this problem. In my case, the server's max_allowed_packet was 1 MB and I could do nothing to change it. And I was inserting some data just above 1 MBs. I found two solution candidates.

1) First, using JDBC. Since MySQL Connector/J v3.1.9, there are a few parameters that you could set, here's my set of parameters in the JDBC URL:

Append these:

blobSendChunkSize=50000&useServerPrepStmts=true&emulateUnsupportedPstmts=false&maxAllowedPacket=20000000

Resulting in JDBC URL:

jdbc:mysql://serverip:3306/databasename?noDatetimeStringSync=true&blobSendChunkSize=50000&useServerPrepStmts=true&emulateUnsupportedPstmts=false&maxAllowedPacket=20000000

Then you must use PreparedStatement to do your inserts, and use the InputStream to pass the byte content as a parameter to setObject. Note that setObject using byte arrays won't enable the blob splitting. The combination of parameters, recent MySQL server (5.0.45 or later), and InputStream will send the blob data using LONG DATA mechanism, splitting the blob according to blobSendChunkSize.

The JDBC solution works and I have tested it.

2) Now, the second candidate, is to use PHP's mysqli driver and use mysqli_send_long_data. For your convenience, copied from PHP manual example :

<?php
$stmt = $mysqli->prepare("INSERT INTO messages (message) VALUES (?)");
$null = NULL;
$stmt->bind_param("b", $null);
$fp = fopen("messages.txt", "r");
while (!feof($fp)) {
    $stmt->send_long_data(0, fread($fp, 8192));
}
fclose($fp);
$stmt->execute();
?>
Leif Gruenwoldt
  • 13,561
  • 5
  • 60
  • 64
YudhiWidyatama
  • 1,684
  • 16
  • 14
  • 1
    Just tried mysqli to store a 27mb file on a 1mb max_allowed_packet. Awesome answer, we've been working on a work around for this for almost a week. – shmeeps Aug 19 '11 at 19:47
  • Isn't is possible to set these parameters just for the current transaction/connection? Maybe using `SET ...`? I would prefer this over changing the JDBC URL. – Tim Büthe Sep 09 '15 at 13:53
  • Worked great when added just these two params useServerPrepStmts=true&maxAllowedPacket=20000000. I saw 'java.sql.SQLException: Parameter of prepared statement which is set through mysql_send_long_data() is longer than 'max_allowed_packet' bytes' which is weird when I used blobSendChunkSize. – Rishi Aug 03 '17 at 23:52
  • I dont familiar about this but i have same trouble. I want to know for first candidate, how to using `blobSendChunkSize=50000&useServerPrepStmts=true&emulateUnsupportedPstmts=false&maxAllowedPacket=20000000` and how to check result like `jdbc:mysql://serverip:3306/databasename?noDatetimeStringSync=true&blobSendChunkSize=50000&useServerPrepStmts=true&emulateUnsupportedPstmts=false&maxAllowedPacket=20000000` – Abdul Aziz Al Basyir Aug 24 '17 at 08:47
  • It's *very important to note* that while this solution does not answer the original question: you can't set the client's `maxPacketSize` larger than the server's `max_packet_size`, so the server always wins. I have not been able to get solutions like these to work with MariaDB 10.3 with JDBC. It seems that you simply cannot insert a field value larger than `max_packet_size`. – Christopher Schultz Sep 16 '20 at 20:09
  • Can someone post a working java/jdbc sample? Having no luck, after trying for many many hours. – cheenu Mar 14 '22 at 16:38
-1

I don't think that there is a way. Maybe splitting the blob will do the trick.

But I think that reading this article http://blogs.msdn.com/b/oldnewthing/archive/2007/03/01/1775759.aspx from Reymond Chen is the best answer to your question.

Relational databases are not designed for that kind of use. If you need to store a gigabyte+ blob in a database, well it is better usually to store it on the NFS and just have a short file location string in your DB. Will save you much troubles down the road.

Daniel Iankov
  • 306
  • 3
  • 7
  • My second question was more a curiosity, the main question is the first one. I don't intend to put a 3GB blob in a single column... and I don't think my first question is related to violating operating system limits. I mean, it would be useful to put a 2MB blob in a database whose max_allowed_packet is just 1MB, when I cannot change this variable... – Leandro Gomide Sep 22 '10 at 18:43