0

I'm trying to get the data from a Vanilla Forums export into SQL Server so I can then write some sort of script to import it into YAF.NET. I've tried with an integration services project and the SQL Server import wizard. The forums and users went over eventually but the topics table is giving me trouble. The problem is the separation of records, I can't get them to split properly with a flat file data source.

Eg:

DiscussionID,CategoryID,InsertUserID,UpdateUserID,Name,Body,Format,CountComments,CountViews,Closed,Announce,Sink,DateInserted,DateUpdated,InsertIPAddress,UpdateIPAddress,DateLastComment,Score
1,2,2,0,"Welcome","","Html",1,1,0,0,0,"2005-11-22 20:36:00","2005-11-22 20:36:00",\N,\N,"2005-11-22 20:36:00",\N
13,5,5,0,"Custom Feilds","Hi Echilon\,\
\
I've been fiddling with iZeit Calendar a bit (though I haven't published anything\, as that would be inapropriate without your permission) and I have tried\, without success\, to add extra fields to both the input form and the calendar output.\
\
So far I've added the extra columns to the database\, attempted to edit the multiple queries in functions.php -> addevent() function with little success.\
\
I was wondering if you could help me out a bit in better understanding the flow of data from input fields to database query.\
\
Once I get the data into the database\, I shouldn't see any future problems of displaying it.\
\
I would also like to note that you've done a fantastic job with this software!\
\ 
-Ax","Html",4,1830,0,0,0,"2006-02-23 00:14:43","2006-02-24 18:57:53",\N,\N,"2006-02-24 18:57:53",\N
3,4,2,0,"[Wallpaper] Aeon Genesis","<a target=\"_blank\" href=\"http://www.deviantart.com/deviation/24402244/\"></a><img src=\"http://mi6.nu/aeon_small.jpg\" border=\"0\" />\
<a target=\"_blank\" href=\"http://www.deviantart.com/deviation/24402244/\">Full Size - 1600x1200</a>\
\
I made this in 2004\, it's an edited photo of the view from my window at sunset.","Html",1,1052,0,0,0,"2005-11-23 09:46:29","2005-11-23 09:46:29",\N,\N,"2005-11-23 09:46:29",\N
4,7,2,0,"Moodsig","This is a script which lets you pick a mood from a control panel\, then have it display that mood in a sig\, along with a random quote from a database. It's not quite finished yet\, but it should be released sometime this week. I just have to track down the guys who made the smileys for permission. It's a bit buggy at the minute\, but it should be working in a few days.\
\
At the minute\, you need PHP and a MySQL database\, but I might release a version which only needs php and a text file if there's demand for it.\
\
This is the control panel\, which controls the sig I'm using now.\
<a target=\"_blank\" href=\"http://mi6.nu/moodsig_0.8.jpg\"></a><img src=\"http://mi6.nu/moodsig_0.8_thumb.jpg\" border=\"0\" />","Html",1,1100,0,0,0,"2005-11-23 15:38:56","2005-11-23 15:38:56",\N,\N,"2005-11-23 15:38:56",\N
5,4,2,0,"Dynasig 1.0","Dynasig lets you set your current mood with a web based control panel\, then display it aswell as a random quote in a signature which you can use on a forum.\
\
<b>Installation</b>\
You'll need a webserver with PHP and MySQL. In the future\, I might release a version that doesn't need MySQL\, but for now\, you have to have it. \
\
1) Set up a database on ...

In this instance, new records start at these lines:

3,4,2,0,"[Wallpaper] Aeon Genesis","<a ta
4,7,2,0,"Moodsig","
5,4,2,0,"Dynasig 1.0","Dynasig l

How would I get this into SQL server?

Echilon
  • 10,064
  • 33
  • 131
  • 217
  • You haven't said exactly what "giving me trouble" means: errors, unexpected results? And what are the column and row delimiters in your sample data? I've often found that SSIS or bcp.exe fail to parse CSV files that have large text fields because of problems with delimiters. See [this Connect issue](https://connect.microsoft.com/SQLServer/feedback/details/312164/flat-file-parser-cannot-import-files-with-embedded-text-qualifiers), for example. It may be easier to do this in an external script using a language with good CSV support (e.g. Perl, Python) and load it that way. – Pondlife Mar 15 '13 at 18:47
  • Sorry, I've added the correct split locations. – Echilon Mar 15 '13 at 19:28

1 Answers1

0

If you're solely dealing with specific delimiters (such as a comma), you can use the BULK INSERT task (see: http://blog.sqlauthority.com/2008/02/06/sql-server-import-csv-file-into-sql-server-using-bulk-insert-load-comma-delimited-file-into-sql-server/).

BULK
INSERT ForumTable
FROM 'c:\ForumFlatFile.txt'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO

If you want to import certain tags and statements (I can't tell from the OP, sorry), I would suggest functions. For instance, depending on how the forum posts are exported, build functions that could parse certain values between tags (for instance, a function that would parse out A HREF tags). Sometimes forum posts can lead and end with certain patterns, and a function could "recognize" these beginning and ending tags and grab the in between values.

Kprof
  • 742
  • 1
  • 8
  • 16