poll: problems with MTRRs?

Mon Mar 16 22:23:47 UTC 2009

On Mon, Mar 16, 2009 at 05:11:01PM -0400, D. Hugh Redelmeier wrote:
> Do I remember that you eventually got a BIOS update that fix the
> problems?

Well there was nothing wrong with the what the BIOS was doing.
Unfortunately running with the ICH9 SATA interface in IDE mode (the only
choice on the ICH9 it seems according to intel) means you can only do
32bit DMA access, which means all buffers for SATA access have to go
through an address below 4GB, which means the kernel starts using bounce
buffers (by default it sets aside 64MB for this).

Unfortunately 64MB isn't enough it seems for some workloads and I started
(on first boot with the extra 4GB added) getting lots of out of bounce
buffer space messages.

Conviniently (for me) asus also makes a board called the P5K-R which is
a P5K with an ICH9R rather than an ICH9.  By flashing the P5K-R bios
onto my P5K (using some force and a command line bios updater) I now
have the options of IDE, AHCI and RAID for the SATA controller.
RAID won't work on an ICH9 of course, but contrary to intel's official
opinion, an ICH9 works fine in AHCI mode.  Intel is just upset that
some people were using their ICH9 in AHCI mode rather than paying for
the more expensive ICH9R (which officially does AHCI and fakeRAID).
My P5K board actually used to have an AHCI option in the BIOS, but a
BIOS upgrade removed it after intel apparently had complained to Asus
about license fees or soemthing or other regarding the ICH9 and AHCI mode.
That's when asus released the P5K-R as well.

In AHCI mode I can do 64bit DMA and hence no more bounce buffers.
Much better now.  The ivtv tv tuner still uses bounce buffers though
but that seems to work fine.

> There can be several MTRR problems.  The one I'm concerned about is
> where the BIOS sets up MTRRs with nested ranges.
> 
> I think you had a problem with some RAM not actually being covered by
> an MTRR and therefore being uncached.  That is clearly a BIOS bug.

Some people have had that, almost always with intel boards.  I have never
bought an intel board and almost certainly never will.  Their chipsets
are great, but their boards and BIOSs are not.

> Note: I didn't say that the BIOS is wrong.  In fact, I don't think
> that it is.  It just doesn't get along with Linux.

And probably wouldn't get along with some older windows versions either.
Of course if it works with XP and Vista they probably rightly don't care.
PATs have been around for a long time and its about time X started using
them instead of the mtrr settings.

> My HP has a motherboard made by ASUSTeK Computer INC (captitalization
> from dmidecode's output).  The BIOS is originally from American
> Megatrends Inc.  I presume that both Asus and HP poked at it.

Could be.  Of course what HP specifies they want is often much less than
Asus would offer on a board with their own name on it.

> The MTRR fixup code was added for 2.6.27.  It prints "before" and
> "after" MTRR values (the author of the code accepted my suggestion
> that the "before" values were worth printing).

Oh OK.  I am still running 2.6.26 due to lirc issues I was having
with 2.6.28.

> My systems run 2.6.27.  The fixup code is not enabled by default.  It
> is compiled-in for Fedora 10 kernels and I presume for Ubuntu Intrepid
> kernels.  To enable it, pass the kernel the "enable_mtrr_cleanup"
> parameter.
> 
> If you don't see anything printed, I suspect that the BIOS didn't
> nest any MTRRs.

Nope mine are quite linear.

> | > I'd like to know what distro you are running (/etc/issue) and
> | > what kernel you are running (/proc/version).
> | > 
> | > I'd like to know what your MTRR's look like.  Could you tell me what
> | > /proc/mtrr says?
> | > 
> | > I hope to be able to help you if you do find these messages in the
> | > dmesg output.
> | 
> | Well certainly newer kernels have tried to deal with some of these bugs,
> | although apparently the real solution is for X to start using PAT rather
> | than MTRR since the 8 entry MTRR is awfully limited given todays ram
> | quantities.
> 
> Right.
> 
> | Well here is the /proc/mtrr of one box with 8GB ram running Debian Etch
> | at the moment with the 2.6.24 etchnhalf kernel (amd64 of course):
> | 
> | reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
> | reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
> | reg02: base=0x100000000 (4096MB), size=4096MB: write-back, count=1
> | reg03: base=0x200000000 (8192MB), size=8192MB: write-back, count=1
> | reg04: base=0x400000000 (16384MB), size=16384MB: write-back, count=1
> | reg05: base=0x800000000 (32768MB), size=32768MB: write-back, count=1
> | reg06: base=0xd0000000 (3328MB), size=  16MB: write-combining, count=1
> | 
> | Strangely it doesn't appear in dmesg.  No idea why.  It only has 8GB ram,
> | but seems to have setup a much larger mtrr range.  X works fine though.
> | System is an IBM X3650.
> 
> The "write-combining" shows that you are running X and that it has
> happily changed MTRR 6 from "uncached".

Yep.  Some desktop systems might thing only giving 3GB (exactly) of ram to
32bit systems would be silly, but hey this is a server.  Simple is better.
Using 2 MTRR entries for the first 3GB, and 4 more for the 4 to 64GB
range is quite efficient.

> There is no overlap here.  But you are right: MTRRs 3, 4, and 5 cover
> address ranges that are probably almost unpopulated.  I say "almost"
> since a certain amount of memory from under the 4G boundary has been
> moved above the 8G boundary to make room for PCI bus space (like the
> video card).

I am sure that with 8GB ram, nothing is above address 12GB.  No need to
cover mtrr to 32GB, but oh well.  It works.

> | Here is my mythtv box with 6GB ram on an Asus P5K board (using a P5K-R
> | bios to get AHCI mode on the ICH9 SATA controller which allows 64bit
> | dma rather than the 32bit dma limit in IDE mode):
> | 
> | reg00: base=0xd0000000 (3328MB), size= 256MB: uncachable, count=1
> | reg01: base=0xe0000000 (3584MB), size= 512MB: uncachable, count=1
> | reg02: base=0x00000000 (   0MB), size=4096MB: write-back, count=1
> | reg03: base=0x100000000 (4096MB), size=2048MB: write-back, count=1
> | reg04: base=0x180000000 (6144MB), size= 512MB: write-back, count=1
> | reg05: base=0x1a0000000 (6656MB), size= 256MB: write-back, count=1
> | 
> | No problem there either.  Running 2.6.26 amd64 kernel with Debian
> | unstable.
> 
> Either you are not running X or you are not using a conventional
> driver or it is unhappy.  None of your MTRRs say "write-combining".
> Perhaps your driver and kernel know how to use PAT.  Any idea what's
> going on here?  Perhaps clues are in /var/log/Xorg.0.log (if you are
> running X).

Strangely there are no complains in the Xorg.0.log at all.

> Your video card's address range is probably covered by reg00 or reg01.
> Their ranges are nested within reg02's range and therefore cannot be
> changed to "write-combining" without a reorganization.

Could be.  Seems a bit odd.

> Here's what my program mtrr-uncover suggests:
> 
> Initial MTRR configuration:
>  2  0x000000000-0x0ffffffff write-back
>          0  0x0d0000000-0x0dfffffff uncachable
>          1  0x0e0000000-0x0ffffffff uncachable
>  3  0x100000000-0x17fffffff write-back
>  4  0x180000000-0x19fffffff write-back
>  5  0x1a0000000-0x1afffffff write-back
> 
> Final MTRR configuration:
>  2' 0x000000000-0x07fffffff write-back
> 50' 0x080000000-0x0bfffffff write-back
> 51' 0x0c0000000-0x0cfffffff write-back
>  3  0x100000000-0x17fffffff write-back
>  4  0x180000000-0x19fffffff write-back
>  5  0x1a0000000-0x1afffffff write-back

-- 
Len Sorensen
--
The Toronto Linux Users Group.      Meetings: http://gtalug.org/
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists