I am trying to measure latency when a packet comes in Rx buffer and is copied to application memory. I am measuring it with this code:
struct timespec start, end;
clock_gettime(CLOCK_REALTIME, &start);
recvfrom(sock, msg, msg_len, 0, &client, &client_addrlen);
clock_gettime(CLOCK_REALTIME, &end);
I know this cannot precisely measure the latency. However, I can calculate average latency by receiving many packets, measuring each one, and calculating them. Is there any method to measure latency more precisely? (e.g., latency = (time when recvfrom() is done) - (time when NIC receives a packet from)
)
For a device and device driver, I am using Mellanox connectx-3 and mlx4_en.
CodePudding user response:
I was able to get an almost precise number with recvmsg()
.
Reference
- The most helpful link (https://stackoverflow.com/a/47329376/5215330)
- Network timestamping (https://www.kernel.org/doc/Documentation/networking/timestamping.txt)
- Linux Precision Time Protocol (http://linuxptp.sourceforge.net/)
Code
I am reproducing the code from the first link. This code is not a ready-to-run, but just a snippet from the working code.
static struct timespec handle_time(struct msghdr *msg) {
struct cmsghdr *cmsg = CMSG_FIRSTHDR(msg);
struct scm_timestamping *ts = (struct scm_timestamping *)CMSG_DATA(cmsg);
return ts->ts[0];
}
...
char ctrl[64];
char *msg = malloc(64);
int val = SOF_TIMESTAMPING_RX_HARDWARE | SOF_TIMESTAMPING_RX_SOFTWARE
| SOF_TIMESTAMPING_SOFTWARE | SOF_TIMESTAMPING_RAW_HARDWARE;
setsockopt(sock_fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));
// user buffer
struct iovec iov = {
.iov_base = msg,
.iov_len = msg_len,
};
// ancillary message header
struct msghdr m = {
.msg_name = &client_addr, // struct sockaddr_in
.msg_namelen = client_addrlen, // socklen_t
.msg_iov = &iov,
.msg_iovlen = 1,
.msg_control = &ctrl,
.msg_controllen = sizeof(ctrl),
};
while (1) {
memset(msg, 0, msg_len);
num_received = recvmsg(sock_fd, &m, 0);
start = handle_time(&m);
clock_gettime(CLOCK_REALTIME, &end);
if (verbose) {
double elapsed_time = time_diff(start, end) / 1000;
total_elapsed = elapsed_time;
count ;
printf("%f us %f us\n", elapsed_time, total_elapsed / count);
}
if (sendto(sock_fd, msg, msg_len, 0, (struct sockaddr *) &client_addr, client_addrlen) < 0) {
perror("\nMessage Send Failed\n");
fprintf(stderr, "Value of errno: %d\n", errno);
}
}
The key point is to use setsockopt()
and recvmsg()
. The key mechanism is when you set an option for a certain socket FD, the kernel will set a timestamp based on timestamp flag. After you set them, if you receive a message with struct msghdr
, the kernel will audit the timestamp in a way of SW or HW. When you look into the data, you would be able to get 3 timestamps. These information can be explained below:
The structure can return up to three timestamps. This is a legacy feature. At least one field is non-zero at any time. Most timestamps are passed in ts[0]. Hardware timestamps are passed in ts[2]. ts[1] used to hold hardware timestamps converted to system time. Instead, expose the hardware clock device on the NIC directly as a HW PTP clock source, to allow time conversion in userspace and optionally synchronize system time with a userspace PTP stack such as linuxptp. For the PTP clock API, see Documentation/driver-api/ptp.rst.
See 2.1 from Documentation/networking/timestamping.txt
for detail.
If you want to see HW timestamp then you need to have a specific HW (refer this comment) and turn its feature with ioctl()
. However, there is a convenient tool called linuxptp
, which does this job.