[GTALUG] Open CAPI Standard

[Nicholas Krause]

On 1/3/20 1:08 PM, Lennart Sorensen wrote:
> On Fri, Jan 03, 2020 at 12:16:53PM -0500, Nicholas Krause wrote:
>> It is but the problem is its hitting the limits currently. PCI Express
>> 5 is near the end of the limits it seems.
> Really?  They have plans for a PCIe v6 next year with double the speed
> again (although it sure sounds like they have to do some complicated
> stuff to get there).
That's the problem is that its hitting end of life. Not sure if it
was PCI v6 or v5 that was the limit or near it. :( It may
scale more but the newer versions run much, much hotter
and use more power. Not sure about the heat issues on
version 6 but its there for SSDs on v4.
>
>> That's debatable if you consider the issues with DMA and GPUs. Or
>> PCI Express not being hard aligned for the bytes it requires and having
>> to all of them when accessing it. The problem is similar to SATA for
>> SSDs in that for a lot of cases its fine but actually as a whole in
>> HPC not just supercomputers, SATA is a bad issue outside of cheap
>> storage. Its also not full duplex which is fun in its own right.
> SATA certainly isn't good for SSDs.  NVMe is much much better.
>
> No idea why SATA was made half duplex while SAS is full duplex.
Its one of the reasons a lot of server people like SAS :).
And yes there is no good reason unless it was a limitation
at the time but that's doubtful.
>
>> Lastly, PCIe has been known to be needed to be replaced at some
>> point being near EOL in terms of scaling. The question is what for
>> both accelerator cards, high end networking and GPUs will be the
>> future. DDR ram has the same issue do to my knowledge.
> Got any links to something showing PCIe has to be replaced?  I haven't
> found any yet.
Its common knowledge for three reasons:
1. The issues with scaling past 100GBs in a lane and PCI express
6 gets there only with extra complexity
2. Latency from and to the bus for DMA or second tier memory
3. Not good common for multi threading environments due to
hardware align not always known when accessing the bus

IBM has decided its the future for POWER and others think
so in HPC. If you want a reason why try encoding and running
out of video ram and having to always go across the PCIe
bus for more memory.  PCI 6 is only 124GB which is only
2 and a half time the speed of version 3 of OpenCAPI.
OpenCAPI if its doubling like PCIe than the next version
will be 100GBs per second at least, maybe faster without
complexity. Its to my knowledge and I could be wrong, able
to hit speeds without complexity of between 200GBs
and 300GBs with better latency.

>> The problem is that OpenCapi did that and ran into bottlenecks
>> as soon as they went over 50GBs per second or version 3 actually.
>> PCIe has serious limitations and just working around them isn't
>> going to solve it. Just hit the limit of ram in a modern GPU and
>> start going across the PCI bus for each frame and you have issues.
> Yes CAPI did run as an extension on PCIe.
>
>> What I'm curious about is why try and avoid what seems to be
>> unavoidable in the need of a new standard to replace PCIe as
>> we did not do it with the connectors when creating type C which
>> is going to replace probably all other USB type connectors at
>> some point.
> Well the USB-C connector may be common but not all ports support every
> protocol.  Some are just USB ports after all.
>
>> Honestly I think it will just tickle down as need be from supercomputers
>> and servers,
> Well we will have to see.  It might happen eventually, if people even
> use desktops at that point.  In a laptop I guess it doesn't matter how
> the GPU is connected since that's only an issue at design time.
> Intel apparently still thinks they can make do with PCIe.  They can be
> pretty stubborn.
PCIe is going to be around I assume until the heat issues are too
much or power issues. Its already there on version 4 but lets
see if they can scale without this becoming more and more of
a issue. Interesting fact is that most PCI 3.0 and 4.0 use the
same wiring actually.

Nick
>