poll: problems with MTRRs?

[D. Hugh Redelmeier]

| From: Lennart Sorensen <lsorense-1wCw9BSqJbv44Nm34jS7GywD8/FfD2ys at public.gmane.org>

| Well there was nothing wrong with the what the BIOS was doing.
| Unfortunately running with the ICH9 SATA interface in IDE mode (the only
| choice on the ICH9 it seems according to intel) means you can only do
| 32bit DMA access, which means all buffers for SATA access have to go
| through an address below 4GB, which means the kernel starts using bounce
| buffers (by default it sets aside 64MB for this).
| 
| Unfortunately 64MB isn't enough it seems for some workloads and I started
| (on first boot with the extra 4GB added) getting lots of out of bounce
| buffer space messages.

| In AHCI mode I can do 64bit DMA and hence no more bounce buffers.
| Much better now.  The ivtv tv tuner still uses bounce buffers though
| but that seems to work fine.

I don't understand that.  Here's what I think I know (subject to
correction):

PCI addresses are 64 bits but most devices only use 32-bit addresses.
On ordinary (non-64-bit) PCI, passing 64-bit addresses takes two
transfers.  I don't know about PCIe.

In x86, the normal approach has been to map the PCI 32-bit address
space to the lower 4GiB of physical address space.  So typically, 3.x
GiB is for RAM and some part of the top 1GiB is for devices.

Hence the need for bounce buffers.  (A replay of what happened when
PCs got more than 16M of RAM and the ISA bus couldn't address it.)
(The Broadcom wireless in one of my notebooks can only addess the
bottom 1G -- odd and undocumented.)

The Athlon 64 architecture introduced an IOMMU which allowed a moderately
crude mapping of the PCI address space onto the memory address space.
The IOMMU was on the CPU chip (that's where the memory controller is
in the Athlon 64).

Somehow Intel didn't notice when they cloned the architecture so the
P4 didn't have an IOMMU.  I presume they fixed this for Intel Core.
But where?  The memory controller is on support chips, not the CPU
chip.

So why can the IOMMU not work for your IDE controller?  Why does it
work when you change the BIOS?  Is there an issue related to
scatter/gather?

| > | Here is my mythtv box with 6GB ram on an Asus P5K board (using a P5K-R
| > | bios to get AHCI mode on the ICH9 SATA controller which allows 64bit
| > | dma rather than the 32bit dma limit in IDE mode):
| > | 
| > | reg00: base=0xd0000000 (3328MB), size= 256MB: uncachable, count=1
| > | reg01: base=0xe0000000 (3584MB), size= 512MB: uncachable, count=1
| > | reg02: base=0x00000000 (   0MB), size=4096MB: write-back, count=1
| > | reg03: base=0x100000000 (4096MB), size=2048MB: write-back, count=1
| > | reg04: base=0x180000000 (6144MB), size= 512MB: write-back, count=1
| > | reg05: base=0x1a0000000 (6656MB), size= 256MB: write-back, count=1
| > | 
| > | No problem there either.  Running 2.6.26 amd64 kernel with Debian
| > | unstable.
| > 
| > Either you are not running X or you are not using a conventional
| > driver or it is unhappy.  None of your MTRRs say "write-combining".
| > Perhaps your driver and kernel know how to use PAT.  Any idea what's
| > going on here?  Perhaps clues are in /var/log/Xorg.0.log (if you are
| > running X).
| 
| Strangely there are no complains in the Xorg.0.log at all.
| 
| > Your video card's address range is probably covered by reg00 or reg01.
| > Their ranges are nested within reg02's range and therefore cannot be
| > changed to "write-combining" without a reorganization.
| 
| Could be.  Seems a bit odd.

My guesses are that you are using the proprietary nVidia driver and
that it uses PAT.  I noticed a contribution from nVidia to the kernel
to support PAT six years ago! http://lkml.org/lkml/2003/5/20/131
I don't actually know if it was adopted then.

Sad that the open source drivers don't yet use PAT.
--
The Toronto Linux Users Group.      Meetings: http://gtalug.org/
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists