[GTALUG] Open CAPI Standard

[Nicholas Krause]

On 1/3/20 11:07 AM, Lennart Sorensen wrote:
> On Thu, Jan 02, 2020 at 11:36:05PM -0500, Nicholas Krause via talk wrote:
>> Greetings,
>> I mentioned this to some of you at the meeting in December but IBM and
>> others are working
>> on creating a new standard to replace PCI express and some of its
>> limitations called Open
>> CAPI. The current versions is as fast as NVLink at 50GBs in a x16 lane from
>> memory which
>> future versions being able to hit over 100GBs in the same lane speed and
>> 250ms latency
>> guaranteed as compared to the 500ms to 750ms in PCI express.
>>
>> This is a link:https://opencapi.org/ to anyone who wants to read the spec,
>> through you will
>> have to give them a email and I believe where you work. Currently POWER is
>> the only
>> real architecture supporting it through or NVlink for that matter.
> In order for that to have any chance of hitting consumer machines it would
> have to be much better than PCIe due to not being backwards compatible.
>
> AGP solved the serious bandwidth limitations of PCI that video cards
> needed, but was also a single slot for a single purpose and didn't
> replace PCI.  PCIe replaced PCI since it was much faster, much easier
> to implement, and had future scalability which PCI couldn't have (a
> parallel shared bus just isn't going to scale).
>
> Given PCIe is improving over time and stays backwards compatible,
> being two the speed or half the latency likely isn't enough to make
> anyone bother.
It is but the problem is its hitting the limits currently. PCI Express
5 is near the end of the limits it seems.
>
> So for special purpose like super computers, both nvlink and opencapi
> have good use cases, but I don't expect to see it in my home PC.  It just
> doesn't add enough versus PCIe to warrent a change.  PCIe isn't dead end
> after all, the way PCI was.
That's debatable if you consider the issues with DMA and GPUs. Or
PCI Express not being hard aligned for the bytes it requires and having
to all of them when accessing it. The problem is similar to SATA for
SSDs in that for a lot of cases its fine but actually as a whole in
HPC not just supercomputers, SATA is a bad issue outside of cheap
storage. Its also not full duplex which is fun in its own right.

Lastly, PCIe has been known to be needed to be replaced at some
point being near EOL in terms of scaling. The question is what for
both accelerator cards, high end networking and GPUs will be the
future. DDR ram has the same issue do to my knowledge.
>
> I do find it interesting that AMD, Google, IBM, Mellanox, nvidia, etc
> but NOT intel are members.
>
> Of course intel is proposing Compute Express Link (CXL) instead, and
> has a few of the opencapi members part of their group too (Like Google
> and HP for example and even AMD).  There is also CCIX which is an
> extension to PCIe to make it batter for super computers and such.
> Given CXL uses PCIe as its base, I actually expect it to have a much
> better chance of a future than opencapi.
The problem is that OpenCapi did that and ran into bottlenecks
as soon as they went over 50GBs per second or version 3 actually.
PCIe has serious limitations and just working around them isn't
going to solve it. Just hit the limit of ram in a modern GPU and
start going across the PCI bus for each frame and you have issues.

What I'm curious about is why try and avoid what seems to be
unavoidable in the need of a new standard to replace PCIe as
we did not do it with the connectors when creating type C which
is going to replace probably all other USB type connectors at
some point.

Honestly I think it will just tickle down as need be from supercomputers
and servers,
Nick

>