[KyleWinfree]

Hi All,
I just bought more ram for my desktop. I had 2gb, and added 2gb more. The motherboard does support 4gb. However, upon rebooting the computer, it only shows 3gb installed. I assume I need to install a different, or update my kernel? The question then, is how. Or should I do something else?
Here is the output of rpm -qa | grep kernel

[awilliamson]

For a start, you need to upgrade your kernel. We're up to 15mdv or 16mdv for 2007.0 now. If the 15mdv version of the stock kernel does not support all your memory, try kernel-enterprise.
_________________
Adam Williamson | Editor, Mandriva Community Newsletter | Mandriva Club Monkey | PR Monkey | Mandriva Bugmaster | Packager | General dogsbody | awilliamson A T mandriva D 0 T com

[PeterDurr]

Is there any good info (in the Wiki perhaps?) on how to go about upgrading to a newer kernel version, Adam?

I think I tried doing it once, but there was some error about some kind of "stripped" kernel (if I remember right?) package needing to be installed first for it to work correctly.

I would like to know how to go about doing it just for the knowledge I would gain.

[KyleWinfree]

K, update to new kernel, done. I installed the sources too, as I figured I needed to install my nvidia driver by hand afterwards. That all went well. Now,

[Germ]

uname -r will tell you your running kernel.

I had to add the mem= parameter to lilo.conf to get 2008 to see all my RAM.

http://forum.mandriva.com/viewtopic.php?t=74313
_________________
the animal rescue site

[Jose Luis_7456]

I had the same problem on another motherboard (Asus P5d). The bios has an option that limits the memory to 3 GB even if you have installed 4 GB. Activating the option for larger memory crashes the kernel shipped with 2007.1 X86_64. So you have to install a new kernel before changing the bios setings.

[drodrig]

However, in my case I don't manage to have all my 4GB ram memory.

I upgraded all my hardware recently, without changing my Mandriva 2007.1 OS. I tried many kernels, in the entreprise series, and I don't manage to see all of my ram. I just tried once more with kernel-enterprise-2.6.17.16mdv-1-1mdv2007.1.i586.rpm and I sill have the following :

free -m
total used free shared buffers cached
Mem: 2026 753 1273 0 72 447
-/+ buffers/cache: 233 1793
Swap: 4094 0 4094

or cat /proc/meminfo
MemTotal: 2075252 kB
MemFree: 1303284 kB
Buffers: 74492 kB
Cached: 458508 kB
SwapCached: 0 kB
Active: 333576 kB
Inactive: 333592 kB
HighTotal: 1179076 kB
HighFree: 569412 kB
LowTotal: 896176 kB
LowFree: 733872 kB
SwapTotal: 4192880 kB
SwapFree: 4192880 kB
Dirty: 660 kB
Writeback: 0 kB
Mapped: 190212 kB
Slab: 80848 kB
CommitLimit: 5230504 kB
Committed_AS: 262216 kB
PageTables: 2988 kB
VmallocTotal: 116728 kB
VmallocUsed: 52044 kB
VmallocChunk: 57332 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB


dmesg gives that
1151MB HIGHMEM available.
895MB LOWMEM available.

However, the bios says at the boot that there is 4096M.

My motherboard is a P5B premium and I have an Intel dual core 2 GHz.

My main question will then be : is this problem due to the fact that my OS was previously installed on a 2GB machine ? Is it necessary to install everything when you do hardware upgrade (new motherboad + new proc + 4GB mem) ?

[martin_whitaker]

Your problem is that you have installed a 32-bit version of the OS, which only supports up to 3.1GB of RAM. You need to install the 64-bit version (x86_64) if you want to use more RAM.

I don't think you can just install a 64-bit kernel - too many other things depend on it, and I certainly wouldn't recommend trying an OS upgrade, so you'll probably need to do a full OS reinstall.

I don't know what the state of the 64-bit version is these days - last time I tried it, it still had a lot of problems. So you may want to monitor your RAM usage for a while and see how much you would really use the extra 1GB before making the decision to reinstall.

[KyleWinfree]

Martin,
I've heard of this issue for windows, but I could have sworn that I have seen other 32bit linux installs with more than 3.1gb of ram. What about multiple core machines? I do have a motherboard capable of using a dual core pentium, maybe that is in order? At that point, maybe 2gb would go to each core? I bought the ram because I have been running out of memory for some mathematical computations in matlab, so I do unfortunately need every little bit I can get.
I'm also not sure I'm ready to try the 64bit solution yet, as last I tried it, I too had more issues than it was worth.
I'll try the lilo mem config mentioned by Germ and get back to everyone.
Thanks to all so far
Kyle
_________________
Remember, anything you dream is possible, given that you care not who gets the credit.

[KyleWinfree]

On second thought, it looks as if the lilo fix is for up to the 3.1gb limit.
Maybe the dual core idea again?
_________________
Remember, anything you dream is possible, given that you care not who gets the credit.

[martin_whitaker]

Kyle,

Yes, there is a way to get access to up to 64GB of physical RAM with a 32-bit Linux install you need a kernel built to use the physical address extension mechanism. I believe the enterprise kernel is built with this option enabled. The downside is that there is a performance hit (I've seen estimates of 3-6%, although that was some time ago - things may have got better), which is why it's not enabled by default. Also, any single process cannot use more than 3GB of virtual memory, regardless of how much physical memory is installed. So if your matlab process is running out of memory (rather than just running slowly because it is paging to disk), extending the amount of physical RAM won't help - you need a 64-bit OS.

All the dual core systems I've come across treat memory as a common resource, so I doubt you'll find one that lets you allocate half the RAM to each core. And anyway, this wouldn't help if your problem is one process needing lots of memory.

Hope this helps,

Martin

[KyleWinfree]

I've tried some of the things outlined above. No dice.
Both the lilo option, and the enterprise kernel don't result in more than the 3.1gb of memory. Other ideas? Thanks for the help!
_________________
Remember, anything you dream is possible, given that you care not who gets the credit.

[KyleWinfree]

On the single process needing more than 3 gb of memory, maybe my swap isn't configured correctly. Aside from top or meminfo, is there a way to test that I'm using my swap when necessary?
_________________
Remember, anything you dream is possible, given that you care not who gets the credit.

[martin_whitaker]

It's possible that your motherboard/BIOS doesn't support PAE. After booting with the enterprise kernel, if you look in /var/log/syslog you should find some messages like this:

Nov 10 19:38:48 wotan kernel: BIOS-provided physical RAM map:
Nov 10 19:38:48 wotan kernel: BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
Nov 10 19:38:48 wotan kernel: BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
Nov 10 19:38:48 wotan kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
Nov 10 19:38:48 wotan kernel: BIOS-e820: 0000000000100000 - 000000003fee0000 (usable)
Nov 10 19:38:48 wotan kernel: BIOS-e820: 000000003fee0000 - 000000003fee3000 (ACPI NVS)
Nov 10 19:38:48 wotan kernel: BIOS-e820: 000000003fee3000 - 000000003fef0000 (ACPI data)
Nov 10 19:38:48 wotan kernel: BIOS-e820: 000000003fef0000 - 000000003ff00000 (reserved)
Nov 10 19:38:48 wotan kernel: BIOS-e820: 00000000f0000000 - 00000000f4000000 (reserved)
Nov 10 19:38:48 wotan kernel: BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
Nov 10 19:38:48 wotan kernel: 126MB HIGHMEM available.
Nov 10 19:38:48 wotan kernel: 896MB LOWMEM available.

What do you get? (search backwards from the end of the file to make sure you're seeing the most recent log entry).

Also check the config file for the enterprise kernel to make sure 64GN support is enabled. You should find it in /boot, and it should contain the following lines:

# CONFIG_NOHIGHMEM is not set
# CONFIG_HIGHMEM4G is not set
CONFIG_HIGHMEM64G=y

I would use top to check if swap was working - run up a few big processes and check that the "swap used" number changes. Again note that swap space will not increase the 3GB limit for a single process.

[RJ549]

On a 32-bit machine with exactly 4GB, I wouldn't recommend using the HIGHMEM64G configuration. Doing that will open a paging window below 4GB in order to access memory above 4GB (of which there isn't very much) and cause a performance hit.

PAE is smoke-'n'-mirrors but before there were 64-bit commodity platforms, it was the only way for large databases to access > 4GB on a 32-bit system. Nowadays, 64-bit servers are the way to go.

The reason you can't see the full 4GB in /proc/meminfo is because your hardware has built-in RAM and non-volatile ROM which also require physical addresses. 4GB (2^32) is the hard limit for physical+device memory on a 32-bit OS.

If you switch to the 64-bit OS, you'll be able to use the full 4GB of physical memory and performance will be *much* better than a 32-bit OS using PAE.
_________________
Club helper, crusader and house-trained primate.

MSI K8N Neo4, Athlon64 3200+, 2GB PC3200 dual-channel, GeForce 6600 PCIe 256M, Lite-On DVD/CD-RW, Memorex DVD-RW, AG Neovo 19" DFP. MDV 2008.0 x86_64.

[martin_whitaker]

RJ:

[KyleWinfree]

There is a lot more to my question than I had originally expected. I'm not sure I'm ready for the dependency hell, as I really appreciate that things just work in the 32 os. So let me see that I understand where I stand at this point:

* I have 4gb of ram installed
* 1gb is taken by the kernel, and used to do things that would otherwise eat into my ram if say, I only had 2gb. This is used by the kernel because it's not easily accessible by the user/system (the 32 bit - 36 bit memory addressing), and might as well be used by something.
* The other 3gb is used by system processes, including my memory hog matlab routines and simulations.
* So I have achieved a performance increase by installing the additional 1gb for kernel operations, and 1gb for user/system operations. Unless I switch to a 64 bit install, or do some tweaking I'm not positive I want to try, this is just how it is and forever will be. Correct?
* Switching to a dual core processor, may, but probably won't help except for multi-tasking and other parallel processes (of which, matlab is not).

I think this is acceptable to me, or at least more so than switching to a 64 bit install. So I suppose the next question is the swap. I have 20gb of hd space dedicated to swap, though I've never really seen it used in the past. Maybe matlab doesn't want to touch it? Is there an easy way to test that the system can, and better yet, does use some of that swap?
Thanks as always, Kyle
_________________
Remember, anything you dream is possible, given that you care not who gets the credit.

[KyleWinfree]

Martin,
Almost forgot to post this: from the enterprise config file:

[RJ549]

[martin_whitaker]

Kyle,

First I should say that I may be raising false alarms about the problems with a 64-bit install - it's 9 months since I tried it, and the problems may have been fixed by now.

Regarding memory and swap usage, the key thing to understand is the difference between virtual and physical memory.

With a 32-bit system, a single user application can address up to 2^32 bytes (4GB) of memory. So the program code and data for that application all have to fit in 4GB. But with a virtual memory system, not all of the 4GB needs to reside in physical memory - code or data that is not currently being used can be stored on the hard disk until it is needed. The space on the hard disk used for this is called "swap space", code or data that is currently stored in physical memory is referred top as being "swapped in", and code or data that is currently stored on the hard disk is referred to as being "swapped out".

The OS detects when an application needs some code or data that is currently swapped out, frees some space in the physical memory by swapping out some code or data that hasn't been accessed recently, then fills that space by swapping in the required code or data.

In a multi-tasking system, each user application can have its own 4GB address space. But that's OK, the OS just has to track which bits of code and data each application is currently using and manage the physical memory usage accordingly (not to make light of it - this is a complex task!).

So in theory you can run as many applications as you like, each using up to 4GB of "virtual" memory. All you need to do is increase the size of your swap partition so there is enough swap space to store them all. In practice it falls down if the amount of swap space greatly exceeds the amount of physical memory - the system ends up spending more time swapping data in and out than it does doing useful work (this is known as "thrashing").

In Linux, the OS has been designed so a portion of each application's virtual address space is mapped to the kernel code and data. This means the same kernel code and data is shared between all applications and allows critical bits of code (like the code which manages the virtual memory) to be locked in physical memory and never swapped out. By default the portion reserved is 1GB - you can change this by building a custom kernel, but it is not recommended to go much below 1GB.

So to summarise:

* A 32-bit OS limits each application's virtual address space to 4GB.
* Linux by default uses 1GB of this, leaving 3GB. This is the maximum any one application can use.
* You can run as many applications as you like provided you have enough swap space. But if too many are running simultaneously, the system may spend all its time swapping rather than doing useful work.
* The amount of physical memory installed affects how frequently code and data are swapped in and out. The more you have, the less time will be wasted in swapping.
* The amount of physical memory has no affect on the 3GB limit for a single application.

Note that when I say application I mean a single running instance of that application. So if you start matlab twice, you will (probably) have two instances running. I say probably because some applications (e.g. Firefox, Acrobat Reader) detect if there is an instance running and if so, share its address space to save having the same code duplicated.

So to conclude:

* Without changing to a 64-bit OS, you can't increase (by much) the 3GB limit for a single matlab session.
* If you did change to a 64-bit OS, but only have a 32-bit version of matlab, the limit will (I believe) go up to 4GB. If you have a 64-bit version of matlab, the limit will be the size of your swap partition.
* At the moment your system seems only to be using 3GB of your physical memory. If you see a lot of hard disk activity when you are running your matlab simulations, it is probably doing a lot of swapping, and it is worth trying to get it to use the extra 1GB. Conversely, if you don't see much hard disk activity, having the extra 1GB won't make much difference.
* As you say, a dual core processor will only be of benefit if you want to multi-task. If you use the same machine for running simulations and doing other work, it's well worth investing in a dual core processor (that's why I've got one).

Hope this makes things a bit clearer. Being an electronic engineer, I do tend to use technical terms and assume everyone understands them!

Martin

[martin_whitaker]

RJ:

My

[RJ549]

Kyle: How much video memory does your graphics card have?

Also: Does your BIOS have a setting for AGP Aperture size?
If so, reducing it to a low value (ie: 32MB) should bring some of your system memory back.
_________________
Club helper, crusader and house-trained primate.

MSI K8N Neo4, Athlon64 3200+, 2GB PC3200 dual-channel, GeForce 6600 PCIe 256M, Lite-On DVD/CD-RW, Memorex DVD-RW, AG Neovo 19" DFP. MDV 2008.0 x86_64.

[martin_whitaker]

Out of curiousity, I decided to run some tests to find out what the performance hit of PAE by itself is. I built a custom kernel with HIGHMEM64G enabled, but otherwise with the same configuration as the 2008.0 desktop kernel.

Building a custom kernel took 42 minutes when using the standard desktop kernel and the same time when using my custom kernel (unfortunately I didn't time it to the second, as I was expecting a bigger difference).

Running a processor/memory intensive task (an FPGA synthesis, place and route) took 29 minutes 41 seconds using the standard kernel and 29 minutes 59 seconds using the custom kernel (averaged over three runs).

So on my system (Asus M2N-SLI, Athlon 64 X2 3800+) and performing this task, the performance hit is 1%. Obviously this is too small a sample to reach any definite conclusion, but it does indicate that enabling PAE is not necessarily that hurtful.

[KyleWinfree]

[KyleWinfree]

Martin,
Your summary all makes sense, and leave me with the feeling that for now, it's good enough. Thanks for all the help.
Kyle
_________________
Remember, anything you dream is possible, given that you care not who gets the credit.