[GTALUG] Open CAPI Standard

[Lennart Sorensen]

On Fri, Jan 03, 2020 at 12:16:53PM -0500, Nicholas Krause wrote:
> It is but the problem is its hitting the limits currently. PCI Express
> 5 is near the end of the limits it seems.

Really?  They have plans for a PCIe v6 next year with double the speed
again (although it sure sounds like they have to do some complicated
stuff to get there).

> That's debatable if you consider the issues with DMA and GPUs. Or
> PCI Express not being hard aligned for the bytes it requires and having
> to all of them when accessing it. The problem is similar to SATA for
> SSDs in that for a lot of cases its fine but actually as a whole in
> HPC not just supercomputers, SATA is a bad issue outside of cheap
> storage. Its also not full duplex which is fun in its own right.

SATA certainly isn't good for SSDs.  NVMe is much much better.

No idea why SATA was made half duplex while SAS is full duplex.

> Lastly, PCIe has been known to be needed to be replaced at some
> point being near EOL in terms of scaling. The question is what for
> both accelerator cards, high end networking and GPUs will be the
> future. DDR ram has the same issue do to my knowledge.

Got any links to something showing PCIe has to be replaced?  I haven't
found any yet.

> The problem is that OpenCapi did that and ran into bottlenecks
> as soon as they went over 50GBs per second or version 3 actually.
> PCIe has serious limitations and just working around them isn't
> going to solve it. Just hit the limit of ram in a modern GPU and
> start going across the PCI bus for each frame and you have issues.

Yes CAPI did run as an extension on PCIe.

> What I'm curious about is why try and avoid what seems to be
> unavoidable in the need of a new standard to replace PCIe as
> we did not do it with the connectors when creating type C which
> is going to replace probably all other USB type connectors at
> some point.

Well the USB-C connector may be common but not all ports support every
protocol.  Some are just USB ports after all.

> Honestly I think it will just tickle down as need be from supercomputers
> and servers,

Well we will have to see.  It might happen eventually, if people even
use desktops at that point.  In a laptop I guess it doesn't matter how
the GPU is connected since that's only an issue at design time.
Intel apparently still thinks they can make do with PCIe.  They can be
pretty stubborn.

-- 
Len Sorensen