MIRRORS/linux-yocto
1
0
Fork 0
mirror of git://git.yoctoproject.org/linux-yocto.git synced 2025-10-22 23:13:01 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity

net: add devmem TCP TX documentation

Browse Source 
Add documentation outlining the usage and details of the devmem TCP TX
API.

Signed-off-by: Mina Almasry <almasrymina@google.com>
Acked-by: Stanislav Fomichev <sdf@fomichev.me>
Link: https://patch.msgid.link/20250508004830.4100853-6-almasrymina@google.com
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
This commit is contained in:
Mina Almasry 2025-05-08 00:48:25 +00:00 committed by Paolo Abeni
parent bd61848900
commit 17af8cc06a
1 changed files with 146 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

150
Documentation/networking/devmem.rst
View File

@ -62,15 +62,15 @@ More Info
https://lore.kernel.org/netdev/20240831004313.3713467-1-almasrymina@google.com/
Interface
=========
RX Interface
============
Example
-------
tools/testing/selftests/net/ncdevmem.c:do_server shows an example of setting up
the RX path of this API.
./tools/testing/selftests/drivers/net/hw/ncdevmem:do_server shows an example of
setting up the RX path of this API.
NIC Setup
@ -235,6 +235,148 @@ can be less than the tokens provided by the user in case of:
(a) an internal kernel leak bug.
(b) the user passed more than 1024 frags.
TX Interface
============
Example
-------
./tools/testing/selftests/drivers/net/hw/ncdevmem:do_client shows an example of
setting up the TX path of this API.
NIC Setup
---------
The user must bind a TX dmabuf to a given NIC using the netlink API::
struct netdev_bind_tx_req *req = NULL;
struct netdev_bind_tx_rsp *rsp = NULL;
struct ynl_error yerr;
*ys = ynl_sock_create(&ynl_netdev_family, &yerr);
req = netdev_bind_tx_req_alloc();
netdev_bind_tx_req_set_ifindex(req, ifindex);
netdev_bind_tx_req_set_fd(req, dmabuf_fd);
rsp = netdev_bind_tx(*ys, req);
tx_dmabuf_id = rsp->id;
The netlink API returns a dmabuf_id: a unique ID that refers to this dmabuf
that has been bound.
The user can unbind the dmabuf from the netdevice by closing the netlink socket
that established the binding. We do this so that the binding is automatically
unbound even if the userspace process crashes.
Note that any reasonably well-behaved dmabuf from any exporter should work with
devmem TCP, even if the dmabuf is not actually backed by devmem. An example of
this is udmabuf, which wraps user memory (non-devmem) in a dmabuf.
Socket Setup
------------
The user application must use MSG_ZEROCOPY flag when sending devmem TCP. Devmem
cannot be copied by the kernel, so the semantics of the devmem TX are similar
to the semantics of MSG_ZEROCOPY::
setsockopt(socket_fd, SOL_SOCKET, SO_ZEROCOPY, &opt, sizeof(opt));
It is also recommended that the user binds the TX socket to the same interface
the dma-buf has been bound to via SO_BINDTODEVICE::
setsockopt(socket_fd, SOL_SOCKET, SO_BINDTODEVICE, ifname, strlen(ifname) + 1);
Sending data
------------
Devmem data is sent using the SCM_DEVMEM_DMABUF cmsg.
The user should create a msghdr where,
* iov_base is set to the offset into the dmabuf to start sending from
* iov_len is set to the number of bytes to be sent from the dmabuf
The user passes the dma-buf id to send from via the dmabuf_tx_cmsg.dmabuf_id.
The example below sends 1024 bytes from offset 100 into the dmabuf, and 2048
from offset 2000 into the dmabuf. The dmabuf to send from is tx_dmabuf_id::
char ctrl_data[CMSG_SPACE(sizeof(struct dmabuf_tx_cmsg))];
struct dmabuf_tx_cmsg ddmabuf;
struct msghdr msg = {};
struct cmsghdr *cmsg;
struct iovec iov[2];
iov[0].iov_base = (void*)100;
iov[0].iov_len = 1024;
iov[1].iov_base = (void*)2000;
iov[1].iov_len = 2048;
msg.msg_iov = iov;
msg.msg_iovlen = 2;
msg.msg_control = ctrl_data;
msg.msg_controllen = sizeof(ctrl_data);
cmsg = CMSG_FIRSTHDR(&msg);
cmsg->cmsg_level = SOL_SOCKET;
cmsg->cmsg_type = SCM_DEVMEM_DMABUF;
cmsg->cmsg_len = CMSG_LEN(sizeof(struct dmabuf_tx_cmsg));
ddmabuf.dmabuf_id = tx_dmabuf_id;
*((struct dmabuf_tx_cmsg *)CMSG_DATA(cmsg)) = ddmabuf;
sendmsg(socket_fd, &msg, MSG_ZEROCOPY);
Reusing TX dmabufs
------------------
Similar to MSG_ZEROCOPY with regular memory, the user should not modify the
contents of the dma-buf while a send operation is in progress. This is because
the kernel does not keep a copy of the dmabuf contents. Instead, the kernel
will pin and send data from the buffer available to the userspace.
Just as in MSG_ZEROCOPY, the kernel notifies the userspace of send completions
using MSG_ERRQUEUE::
int64_t tstop = gettimeofday_ms() + waittime_ms;
char control[CMSG_SPACE(100)] = {};
struct sock_extended_err *serr;
struct msghdr msg = {};
struct cmsghdr *cm;
int retries = 10;
__u32 hi, lo;
msg.msg_control = control;
msg.msg_controllen = sizeof(control);
while (gettimeofday_ms() < tstop) {
if (!do_poll(fd)) continue;
ret = recvmsg(fd, &msg, MSG_ERRQUEUE);
for (cm = CMSG_FIRSTHDR(&msg); cm; cm = CMSG_NXTHDR(&msg, cm)) {
serr = (void *)CMSG_DATA(cm);
hi = serr->ee_data;
lo = serr->ee_info;
fprintf(stdout, "tx complete [%d,%d]\n", lo, hi);
}
}
After the associated sendmsg has been completed, the dmabuf can be reused by
the userspace.
Implementation & Caveats
========================