[buug] 4 gig limit

[Michael Paoli]

Quoting John de la Garza:

> I have a 32bit box running linux.  It has 8 GB.   I understand in a 32
> bit address space you can't address of 4 GB.  Since linux uses virtual
> memory I have read that one process can access up to 4 GB.

Well, if an unsigned 32 bit integer is used, 32 bits can
address up to 4 GiB
2^32=2^2*2^30=4 GiB
... presuming we're addressing bytes (one gets to address 2^32 of
whatever it is one's addressing).
Whether or not it's actually implemented that way may be another
matter (e.g. a signed integer type may be used to allow relative
addressing from any given position - that would cut the useful range
essentially in half - e.g. such is the case with the 2^31-1 byte limit
(at least historically) on UNIX/LINUX files (lseek(2) seeks in
positive or negative relative to any given position in the file, and
it must also be able to seek zero bytes from any given offset, so one
would then end up with a maximum file size of 2^31-1 bytes, or 1 byte
short of 2 GiB).

x86 architectures (at least >=i386) tend to have various 32 bit
limits, so one typically encounters boundaries around 2 and 4 GiB.
Some hardware architectures may further limit that (e.g. actual
address lines may be less than what the CPU could otherwise address).
Some of the newer, higher end x86 chips break at least many of those
n GiB (where n is a small integer) boundaries, ... but not
necessarily all of them, and some such limits may exist in the
kernel and/or other software (or one might needs to set specific
flags/options, etc. to generally or specifically overcome at least
some, if not many, of those limits).

I'm no exactly an expert on the latest and greatest x86 chips and
their architecture, and various LINUX limits related to such ...
perhaps someone else can fill in some more of those details or provide
a handy suitable reference.

For a quick and dirty check, here's an older C program I wrote ... it
basically does a quick divide and conquer approach to see how much
memory a program (process) can grab (malloc(3)), and then reports
that.  For relatively large amounts of memory relative to the
footprint of the process overhead, it's probably also fairly useful
for approximating how large the process can get.  One might also
potentially modify the program (e.g. delaying its freeing of memory
and exit) to be able to then check on the size of the process by
other means - note, however, that if it's gobbling up all the memory
available on the system, one might not want to leave it in that state
very long (that's also one of the reasons I have it free the memory
right after it's malloc(3)ed it.).  It can also be useful for
checking that one has various resource limits (e.g. ulimit, etc.) set
as one desires.  No guarantees that one won't need to tweak it to
deal with current applicable data sizes/types.

$ ls -lon memgrab.c
-rw-------    1 1003          919 Nov  1  1998 memgrab.c
$ expand -t4 memgrab.c
#include    <stdio.h>
#include    <stdlib.h>
main()  {
    size_t  l,
            m,
            h;
    void    *p;

    l=(size_t)0;    /*  l<=memory we can grab   */
    h=(size_t)1;    /*  h>=memory we can grab   */

    /*  find an h that's too big    */
    while((p=malloc(h))!=(void *)NULL) {
        free(p);
        l=h;    /*  we got this much fine   */
        h=(size_t)2*h;  /*  try this next   */
        if((size_t)h<(size_t)l) {   /*  did we overflow?    */
            fprintf(stderr,"overflow:
(size_t)2*(size_t)%lu<(size_t)%lu\n",(unsigned long)l,(unsigned long)l);
            fflush(stderr);
            exit(1);
        }
    }

    /*  continue until h-l==1   */
    while((size_t)h-(size_t)l>(size_t)1)    {
        /*  we carefully construct our "average" (middle) to avoid
            possible overflow
        */
        m=(size_t)l+((size_t)h-(size_t)l)/(size_t)2;
        if((p=malloc(m))!=(void *)NULL) {
            free(p);
            l=m;    /*  we got this much fine   */
        }
        else    {
            h=m;    /*  couldn't get this much  */
        }
    }
    printf("%lu\n",(unsigned long)l);
    fflush(stdout);
    exit(0);
}
$ 

> I read to use this on systems with more than 4 GB of physical ram you
> must have PentiumPro with PAE (Physical Address Extension).
> [root at hela init.d]# cat /proc/meminfo  | head -n2
> MemTotal:      2074876 kB
> MemFree:       1270664 kB
>
> here is my cpu info
> [root at hela init.d]# cat /proc/cpuinfo  | grep name
> model name      : Intel(R) Pentium(R) III CPU family      1400MHz
> model name      : Intel(R) Pentium(R) III CPU family      1400MHz
>
> I was told these are Xeon chips.  Are Xeon classified as Pent III?
> and do these chips have PAE?
>
> I guess my ultimate question is can I use the 8GB of ram I have at
> least at a per process level?