Skip to main content.
home | support | download

Back to List Archive

Re: Again: Indexing laaarge ammounts of data (more than

From: David L Norris <dave(at)not-real.webaugur.com>
Date: Wed Sep 24 2003 - 20:31:56 GMT
On Wed, 2003-09-24 at 06:23, Peter Asemann wrote:
> Is that 2 GB limit so hard-coded that it's virtually impossible to remove?
> Or are there places, libraries, modules where replacing some code and types 
> could do wonders? If there's still that 2 GB limit and anyone is interested
> in breaking it I'd like to offer my support in programming and testing a 
> large-file version of swish-e.

It's more of an operating system limitation than a SWISH-E limitation. 
On Unix it can be as simple as recompiling with a few extra definitions
to enable 64-bit file support.  Although, there could be hidden integer
problems if types are assumed in SWISH-E (quite likely).  However, type
problems should be corrected anyway.

Here are a couple of places with "large file" info:
  http://ftp.sas.com/standards/large.file/
  http://www.faqs.org/faqs/arch-storage/part2/section-121.html

But, fixing SWISH-E alone won't allow you to access files larger than 2
GB.  The primary problem is that 32-bit filesystems don't support files
larger than 2 GB.  IBM, Sun, SGI, BSD, Apple, Linux, etc systems support
64-bit files.  However, you'd need to use a 64-bit filesystem like XFS
(SGI or Linux) to have files larger than 2 GB.  Linux ext2/ext3, for
example, can only support 2 GB files (maybe 4 GB?).  (Also note that
most NFS implementations are limited to 32-bit.)

Windows may be a lost cause since Microsoft doesn't seem to have any
64-bit file compatibility with Unix.  It should be possible to
reimplement the Unix 64-bit file functions as a separate library on
Windows but it's a ton of extra work.  And, as I read it, it may need to
be implemented differently between Win32 and Win64.

-- 
 David Norris
  http://www.webaugur.com/dave/
  ICQ - 412039
Received on Wed Sep 24 20:32:23 2003