0

I am writing a crawler atm and i am saving crawled pages source data at the database as a nvarchar(max)

As you can guess this is taking huge space so i believe it would be better apporach for me to compress HTML source at the C# - then save it at the database and when fetching back de-compress it

How can i do this in a most effective way ?

I checked and it seems like there is no way to achieve this at sql server

Ty very much for answers.

c# .net 4.5 wpf

sql server 2014

Furkan Gözükara
  • 22,964
  • 77
  • 205
  • 342
  • So I guess you don't need to search within the database for that source text? And what do you mean, "no way to achieve this at sql server"? Why don't you just use `varbinary(max)` if you want to store compressed bytes? – Kirk Woll Sep 14 '14 at 14:10
  • @KirkWoll do you think c# compression or varbinary compression which one better ? – Furkan Gözükara Sep 14 '14 at 14:21
  • @MonsterMMORPG Varbinary isn't compression. It's a binary data type. If you're using SQL Server 2012 or above, you should look forward for File Tables. – Matías Fidemraizer Sep 14 '14 at 14:23
  • @MonsterMMORPG, the former would require the latter. – Kirk Woll Sep 14 '14 at 14:24

1 Answers1

2

If you want to compress or decompress in C#, there are a lot of commercial or free libraries as well.

Or you can use the built-in classes in System.IO.Compression namespace.

Biri
  • 7,101
  • 7
  • 38
  • 52