[Lilug] Poor mans ETL processing

[Peter Capazzi]

I'm not sure... some kind of a non-disclosure agreement between our
company and the vendor. They told me this from THE BEGINNING. (Side
note: Anybody know Latin?) ;-)  That never made sense to me... does that
mean I can't list the application on my resume?
 
CSV's... delimited and fixed-width. I was going to try my hand at
writing a process or two in a bash script. I was just wondering if
anyone ever saw a GUI or even ncurses front end for this. 
 
Some of the parameters for the functions in the applcation are the same
as the command switches, max buffer for sorts, temporary directory to
use, etc. 
 


Peter, 

Whose stopping you from mentioning it? Employer or vendor?

Also what are you ETLing from? CSV, EDI, XML, etc.

At a basic level though, you can write shell scripts pipe all those
commands to get your data in a format that can be bulkloaded into your
database..



On Tue, Feb 24, 2009 at 3:21 PM, Peter Capazzi
<Peter.Capazzi at tranzactis.com> wrote:


	At our work place we use a software that has a pretty heavy
price tag. I'm actually not supposed to mention it by name. Anyway, the
more I work with and the more I learn about linux I see some
similarities between commands in linux and the functions in this
application. 
	 
	I was wondering if there was a "poor man"s ETL engine that is
simply a front end to commands like these:
	sort - sorting datasets on a field
	join - merging data sets
	cut - field selections/exclusions
	cat - for input files
	sed - To modify fields
	grep - To filter
	awk - To modify fields and partition datasets. 
	 
	 

	_______________________________________________
	Lilug mailing list
	Lilug at lilug.org
	http://lists.lilug.org/listinfo.cgi/lilug-lilug.org
	
	


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lilug.org/pipermail/lilug-lilug.org/attachments/20090224/160f733d/attachment-0003.htm>