Monday, February 20, 2012

Merging, and loading dynamic flat file.

I have number of csv files in a folder, all of them with same columns, need to be merged into one table and imported to sql server.

-The first row of the csv file is a header.
-The csv files are updated everyday
-The destination table is replace by new table with new info in the csv.
-The new csv files can be created and old csv files may no longer exist, but we are only interested in information contain in current csv files in the folder.
-I need SSIS to combine all the csv files in the folder and merge into one table.
-Other issue is that the field names may change in csv, so can the SSIS package recognize the change in field name and made necessary change in destination table as well.

Any insight on this issues will be greatly appreciated?

Put simply this is not a scenario that SSIS supports. The key problem is the dynamic nature of the file structure, which SSIS has no run-time support for. The target scenarios for SSIS revolve around unattended/automated data transfer, and any kind of dynamic format is just never going to work well with SSIS.

What use would a new table be that the system has never seen before? For you, you obviously then interpret this, but unless you tell SSIS to do that up front, it would never make sense for it.

No comments:

Post a Comment