Sunday, March 14, 2010

Infobright ICE 3.3.2 Beta released

Infobright ICE 3.3.2 Beta has been released. It adds full support for storing and querying UTF-8 data. The release notes can be found here, and ICE 3.3.2 Beta can be downloaded here.
As this is a beta it should be used in production systems yet, but I can only recommend to try it out if you need UTF-8 support.
As I did the last time I took a look at the differences between the two latest version, 3.3.1 and 3.3.2 Beta.
Most are UTF-8 stuff, but there are couple of other interesting changes which isn't fully implemented in the beta.

It looks like a new configuration parameter are about to be added, UseMySQLImportExportDefaults. It is not implemented yet, meaning it won't do anything different if enabled in the configuration file. But it sounds like it will use the default MySQL import and export settings, instead of the current Infobright settings.

It also looks like a lot of other export/import improvements are under way. For example
  • Ignore lines
  • Load data local infile
  • Value list elements
  • Lock options
  • Optionally enclosed
  • Lines starting by
Some of these options may never be implemented in Infobright, but at least now you get a warning if the options are used, saying that they aren't supported and will be ignored. But the fact that they now appear in the source code, are a step in the right direction.

There have also been some improvements to sorting, and the caching mechanisms.

Monday, March 1, 2010

ICE LOAD DATA LOCAL INFILE

Currently no version of ICE supports the LOAD DATA LOCAL INFILE statement. Although some ETL tools provide a workaround for, see http://www.infobright.org/Forums/viewthread/1123/, by using named pipes and a program installed on the server.
Those who don't use any ETL tool, cannot do anything else than copying the file to the server and load in from there.
I've created a patch which enables remote data loading, but currently it only works on Linux. It works by using named pipes.
When I have more time I'll make it to work on Windows too.

Patch have to applied in the root of Infobright source code, and are tested in ICE 3.3.1 on Linux, it may work on other UNIX like systems, but it won't work on Windows.

Update:
I have discovered a bug in the original patch. The bug allows the data loading to end prematurely, meaning that all rows aren't going to be imported. I've created a temporary fix, until I figure out why this happens.
The fix can be downloaded here.

Patch file:
loadDataLocal331.patch (9 KB)