They are secretive. They are suave. They are a mystery.

Call it a hobby. A company that acts like it is operating in a world of cold war spy agencies rather than the slightly dry and uneventful theatre of database extraction is almost bound to make one curious. On one forum I recently saw a consultant wail with indignation, saying they would not even respond to a potential sale before he signed a non-disclosure agreement. On another I read a researcher declare that he would be excluding them from his market overview for the very same reason. They won’t talk, they won’t show, and if you do talk about them there is always the possibilty that there will be men in matching raincoats coming to have a friendly, private, and potentially terminal chat with you. One cannot but stop and wonder: why? Why, for goodness’ sake, would a company doing business in a competitive market want to act this way? What’s the point of it?

As it turns, there really is a reason. Or at least an explanation. I was most pleased when I found this article, titled “The Rise and Fall of Thinking Machines” from 1995. It makes for good reading.

(more…)

(more…)

Not that it is unfair of them to ask. I mean, let’s face it: the Business Intelligence projects that we work on generally come with price tags that would enable you or me to spend the rest of our lives on pearly white beaches, were the money to be deposited on our bank accounts directly. And while we’re at it, let’s double face it: a substantial amount of that money - up to 75 % I’ve heard mention - is spent not on the reports that are the end result, but on building the data warehouse and the ETL that is used to fill it with juicy numbers. And to top it all off, let’s triple face it: a whole lot of those data mappings we were building in that time really weren’t all that complicated. So the question, at least, is a valid one.

(more…)

A mathematician considers profiling the data as an easy job: only key statistics (frequencies, averages) are derived and no high level mathematics is involved. I remember the face of a mathematician, who after having explained what data profiling was, asked the question: what the hell are talking about? Data profiling is of more interest to the analyst, who wants to know if the data comply to buisiness rules and who would like to get more acquintance with the data.

Data profiling is mostly done during the analysis of an ETL project when data are to derived from sourcing systems and to be put in a data warehouse. Before building starts, we need to know the likely content of the data. Based on this knowledge, one could establish a dataflow that takes into account the particuliarities of the data. Example: if we know from the analysis that dataes of birth are subject to human error, we might include a check to see if the date of birth is within an acceptable range.

Previously such analysis was done thru a series of queries, in order to calculate the key statistics for every attribute in the database. Nowadays alternatives haven arisen that make life less complicated. I will mention 3 such alternatives:

In the Oracle Warehouse Builder, a wizard exists that creates a nice overview for an external table that is imported in OWB.

The SQL Server 2008 also includes a wizard to do the data profiling.

Lastly freeware tool exists that can be downloaded from “arrah technologies”. It has a fully functional version that can be used for a limited period. After that, it should be  bought. To install, one should have the exact version of java that is required (1.5.0_09). Also, the way to get it started is somewhat tricky: it will start with “java Profiler”, which is not documented. In their tutorial, they showed a different command line, which did not get it running. Finally the data base connection should be set manually, which can also be tedious. But after all that, we have an intuive profiling tool that does what it should do: give insight into the data by means of key statistics.

In this article, I used much information from Henk Jan te Brake. Thanks to him; errors in this short overview are mine!

(more…)