MPTCP Fast Open in Linux kernel

This set of patches will bring “Fast Open” Option support to MPTCP in linux kernel. The aim of Fast Open Mechanism is to eliminate one round trip time from a MPTCP conversation by allowing data to be included as part of the SYN segment that initiates the connection.

IETF RFC 8684: Appendix B. TCP Fast Open and MPTCP. IETF RFC LINK

These new patches (LWN | ML: mptcp.lists.linux.dev) provide the full support of TFO with MPTCP,
which means:

– MSG_FASTOPEN sendmsg flag support (patch 1/8)
– TFO support for the listener side (patches 2, 3, 4, 5/8)
– TCP_FASTOPEN socket option (patch 6/8)
– TCP_FASTOPEN_KEY socket option (patch 7/8)
– SELFTESTS (patch 8/8)

2 methods to Request a cookie (in MPTCP) are currently exist:

1) Listener side + Initator side with: MSG_FASTOPEN
LWN publication: https://lwn.net/Articles/916023/
2) Initiator side with: TCP_FASTOPEN_CONNECT
LWN publication https://lwn.net/Articles/909590/

Code examples for initiator side:
MSG_FASTOPEN (Updated on 6 Nov. 2022)
Here is example of Initiator in userspace program written in C:

#include <arpa/inet.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/socket.h>
#include <unistd.h>
#include <netinet/tcp.h>
#include <string.h>
#include <signal.h>

#define SERVER_PORT 7003
#define MAXLINE 999

int main(int argc, char *argv[])
{
	char sendline[MAXLINE + 1], recvline[MAXLINE + 1];
	char *fileLocation = "./fileToDownload";
	char *serverAddress = "10.10.0.2:1234"
	int sock_fd = -1;
	int ret;
	/* Generate HTTP GET into variable sendline*/	
	snprintf(sendline, 999, 
         "GET %s HTTP/1.0\r\n"   
         "Host: %s\r\n"     
         "Content-type: application/x-www-form-urlencoded\r\n"
         "Content-length: %d\r\n\r\n"
         "%s\r\n", fileLocation, serverAddress , (unsigned int)strlen(""), "");



	
	struct sockaddr_in daddr;
	memset(&daddr,0,sizeof(daddr));      
	inet_pton(AF_INET, "10.10.0.2", &daddr.sin_addr);
	daddr.sin_family = AF_INET;
	daddr.sin_port = htons(SERVER_PORT); 	

	/* set the mprotocol MPTCP */
	sock_fd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);
 
	/* set the MSG_FASTOPEN */
	ret = sendto(sock_fd, sendline, strlen(sendline), MSG_FASTOPEN,(struct sockaddr *) &daddr, sizeof(daddr));

	printf ("info: %d \n", ret);	
	close(sock_fd);
	return EXIT_SUCCESS;

}

PacketDrill test: Link to Github
PacketDrill test result: Link to Github

TCP_FASTOPEN_CONNECT

Here is example of Initiator in userspace program written in C:

#include <arpa/inet.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/socket.h>
#include <unistd.h>
#include <netinet/tcp.h>
#include <string.h>
#include <signal.h>

#define SERVER_PORT 7003
#define MAXLINE 999

int main(int argc, char *argv[])
{
	char sendline[MAXLINE + 1], recvline[MAXLINE + 1];
	struct sockaddr_in daddr;
	int sock_fd = -1;
	int enable = 1;
	int ret;

	snprintf(sendline, 999, 
     "GET %s HTTP/1.0\r\n"  // POST or GET, both tested and works. Both HTTP 1.0 HTTP 1.1 works, but sometimes 
     "Host: %s\r\n"     // but sometimes HTTP 1.0 works better in localhost type
     "Content-type: application/x-www-form-urlencoded\r\n"
     "Content-length: %d\r\n\r\n"
     "%s\r\n", "/prepare2.sh", "6.6.6.6:666" , (unsigned int)strlen(""), "")
	
	memset(&daddr,0,sizeof(daddr));      
	inet_pton(AF_INET, "6.6.6.6", &daddr.sin_addr);
	daddr.sin_family = AF_INET;
	daddr.sin_port = htons(SERVER_PORT); 	
	
	/* set the mprotocol MPTCP */
	sock_fd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);
 
	/* set the TCP_FASTOPEN_CONNECT */
	setsockopt(sock_fd, SOL_TCP, TCP_FASTOPEN_CONNECT, &enable, sizeof(enable)); 
	connect(sock_fd, (struct sockaddr *) &daddr, sizeof(daddr)); 
	ret = sendto(sock_fd, sendline, strlen(sendline), 0,(struct sockaddr *) &daddr, sizeof(daddr));
	
	printf ("info: %d \n", ret);
	close(sock_fd);
	return EXIT_SUCCESS;

}

Environment configuration (linux) for TCP_FASTOPEN:

sudo modprobe veth
sudo ip netns add server
sudo ip netns add client
sudo ip link add veth0 type veth peer name veth1
sudo ip link set veth1 netns server
sudo ip link set veth0 netns client
sudo ip netns exec client ip a a 10.10.0.1/24 dev veth0
sudo ip netns exec server ip a a 10.10.0.2/24 dev veth1
sudo ip netns exec client ip link set dev  veth0 up
sudo ip netns exec server ip link set dev  veth1 up

sudo ip netns exec server bash -c "echo 3 > /proc/sys/net/ipv4/tcp_fastopen"
sudo ip netns exec client bash -c "echo 3 > /proc/sys/net/ipv4/tcp_fastopen"

Versions and their details are presented below:

_PATCH v2_ includes client-server partial support + more impact on existing TCP and MPTCP code. (Expand me)

1. MPTCP cookie request from client.
2. MPTCP cookie offering from server.
3. MPTCP SYN+DATA+COOKIE from client.
4. subsequent write + read on the opened socket.

This patch is Work In Progress and an early draft.

The patch is build against 5.16.0-rc8 net-next.

The patch is located in the next mailing list: https://patchew.org/MPTCP/[email protected]/

The pcap dump:

_PATCH v4_ client-server partial support + less impact on existing TCP and MPTCP code. (Expand me)

The differences between previous versions and v4:
1. An attempt to reduce impact on existing TCP code.
2. 2 files related to mptcp_fastopen are created(*.h + *.c).
3. “subflow_v4_conn_request” is used to call “mptcp_conn_request”(located in “mptcp_fastopen.c”) to process the received packet on the listener side when “SYN” is received during 3way handshake.
4. This chain adds “skb” to “&msk->sk_receive_queue” (“subflow_v4_conn_request”->”mptcp_conn_request”->”mptcp_try_fastopen”->”mptcp_fastopen_create_child”->”mptcp_fastopen_add_skb”)
5. Some minor comments from mailing list are not yet included in the current version of the PATCH.

The patch is located in the next mailing list: https://lore.kernel.org/mptcp/[email protected]/

You can download a TCPDUMP pcap from this link: Download tcpdump pcap

Thoughts on RST issue:
With gdb I got a backtrace for “tcp_send_active_reset”:

We can see here that it executed from “subflow_check_data_avail” (“/net/mptcp/subflow.c”)

We now check the “subflow_check_data_avail” line 1260 and in code snippet#2 see that it is located in the fallback “section” that leads us to possible lines, that lead us in code snippet#1 -> MAPPING_INVALID or MUPPING_DUMMY.

Further investigation of MAPPING INVALID seems to lead us to line 1081 of code snippet #3.

Backtrace:


#0  tcp_send_active_reset (sk=sk@entry=0xffff88800461e080, priority=priority@entry=2592) at net/ipv4/tcp_output.c:3437
#1  0xffffffff81783a9a in subflow_check_data_avail (ssk=0xffff88800461e080) at net/mptcp/subflow.c:1260
#2  mptcp_subflow_data_available (sk=sk@entry=0xffff88800461e080) at net/mptcp/subflow.c:1292
#3  0xffffffff81783f59 in subflow_data_ready (sk=0xffff88800461e080) at net/mptcp/subflow.c:1375
#4  0xffffffff8165c55a in tcp_data_ready (sk=sk@entry=0xffff88800461e080) at net/ipv4/tcp_input.c:4983
#5  0xffffffff8165d93d in tcp_data_queue (sk=sk@entry=0xffff88800461e080, skb=skb@entry=0xffff888003e9b4e0) at net/ipv4/tcp_input.c:5057

code snippet#1

...
                if (unlikely(status == MAPPING_INVALID))
                        goto fallback;

                if (unlikely(status == MAPPING_DUMMY))
                        goto fallback;
...

code snippet#2

fallback:
        if (!__mptcp_check_fallback(msk)) {
                /* RFC 8684 section 3.7. */
                if (subflow->send_mp_fail) {
                        if (!READ_ONCE(msk->allow_infinite_fallback)) {
                                ssk->sk_err = EBADMSG;
                                tcp_set_state(ssk, TCP_CLOSE);
                                subflow->reset_transient = 0;
                                subflow->reset_reason = MPTCP_RST_EMIDDLEBOX;
                                tcp_send_active_reset(ssk, GFP_ATOMIC);
                                while ((skb = skb_peek(&ssk->sk_receive_queue)))
                                        sk_eat_skb(ssk, skb);
                        } else if (!sock_flag(ssk, SOCK_DEAD)) {
                                WRITE_ONCE(subflow->mp_fail_response_expect, true);
                                sk_reset_timer((struct sock *)msk,
                                               &((struct sock *)msk)->sk_timer,
                                               jiffies + TCP_RTO_MAX);
                        }
                        WRITE_ONCE(subflow->data_avail, MPTCP_SUBFLOW_NODATA);
                        return true;
                }

                if (!subflow_can_fallback(subflow) && subflow->map_data_len) {
                        /* fatal protocol error, close the socket.
                         * subflow_error_report() will introduce the appropriate barriers
                         */
                        ssk->sk_err = EBADMSG;
                        tcp_set_state(ssk, TCP_CLOSE);
                        subflow->reset_transient = 0;
                        subflow->reset_reason = MPTCP_RST_EMPTCP;
line 1260 >>            tcp_send_active_reset(ssk, GFP_ATOMIC);
                        WRITE_ONCE(subflow->data_avail, MPTCP_SUBFLOW_NODATA);
                        return false;
                }

                __mptcp_do_fallback(msk);
        }

...

code snippet #3

...
        if (subflow->map_valid) {
1067
                /* Allow replacing only with an identical map */
1068
                if (subflow->map_seq == map_seq &&
1069
                    subflow->map_subflow_seq == mpext->subflow_seq &&
1070
                    subflow->map_data_len == data_len &&
1071
                    subflow->map_csum_reqd == mpext->csum_reqd) {
1072
                        skb_ext_del(skb, SKB_EXT_MPTCP);
1073
                        goto validate_csum;
1074
                }
1075

1076
                /* If this skb data are fully covered by the current mapping,
1077
                 * the new map would need caching, which is not supported
1078
                 */
1079
                if (skb_is_fully_mapped(ssk, skb)) {
1080
                        MPTCP_INC_STATS(sock_net(ssk), MPTCP_MIB_DSSNOMATCH);
1081
                        return MAPPING_INVALID;
1082
                }
1083

1084
                /* will validate the next map after consuming the current one */
1085
                goto validate_csum;
1086
        }

_PATCH v5_ client-server partial support + Multiple improvements. (Expand me)

Source code:
Link to the mailing list of the linux upstream mptcp
or
Link to the PATCHWORK of the linux upstream mptcp

===Changes between v4 and v5
-All data sent by the initiator received by listener in userspace.
-No more infinite re-transmissions of the same packet in the end of the second connection.
-”WARNING in sk stream kill queues”, when skb is added to msk queue – is fixed.
-Fix the second (discarded) packet that is re-transmitted in the end of 2nd session.
-We have (in the v4 it is not present) the FIN in the end of the 2nd (TFO data+cookie+syn) session.
===Tradeoffs of v5:
-Minimal impact on existing TCP code in linux kernel.
-Icreased size of the patch. Some functions are inspired from existing code in fastopen.c and others.
===Future work:
Address the appearance of ”MPTCP FIN” as duplicated acks.

_PATCH v6_ client-server partial support + Multiple improvements. (Expand me)

Source code:
Link to the mailing list of the linux upstream mptcp

===Changes between v5 and v6

– Change commit length
– Removed rmem commit (refactor)
– fixed setsocketopts
– separated helpers from add_skb_to_msk_fastopen into different patch.
===Tradeoffs of v5:
-Minimal impact on existing TCP code in linux kernel.
-Icreased size of the patc,
===Future work:
Address the appearance of ”MPTCP FIN” as duplicated acks.

_PATCH v7_ client-server partial support + Multiple improvements. (Expand me)

Source code:
Link to the mailing list of the linux upstream mptcp

[PATCH v7] includes “client-server” partial support for:
1. MPTCP cookie request from client (seems OK).
2. MPTCP cookie offering from server (seems OK).
3. MPTCP SYN+DATA+COOKIE from client (seems OK).
4. subsequent write + read on the opened socket (seems OK).

===Changes between v6 and v7
– Order of commits is changed.
– Code is refactored
===Pros/Cons of v7:
– little modifications of existing TCP code in linux kernel.
– relatively decreased size of the patch (in comparison with v6)
due to reuse of some functions in tcp_fastopen.c.
===Future work:
-Adress the appearance of “MPTCP FIN” as duplicated acks.

_PATCH v8_ client-server partial support + Refactored code (beginning). (Expand me)

Link to the mailing list of the linux upstream mptcp

[PATCH v8] includes “client-server” partial support for:
1. MPTCP cookie request from client (seems OK).
2. MPTCP cookie offering from server (seems OK).
3. MPTCP SYN+DATA+COOKIE from client (seems OK).
4. subsequent write + read on the opened socket (seems OK).
===Changes between v7 and v8
– We change from 0impact approach on existing TCP code to
full reusage of existing and available functions.
– Code is refactored (Max. reuse of existing linux kernel code).
– fastopen.c is reduced to minima.
– Other comments from mailing list are coming in the next version.
===Future work
-Adress the appearance of “MPTCP FIN” as duplicated acks.
-Integrate presented in the last patch selftests.

_PATCH v9_ client-server partial support + advanced refactoring. (Expand me)

Link to the mailing list of the linux upstream mptcp

[PATCH v9] includes “client-server” partial support for:
1. MPTCP cookie request from client (seems OK).
2. MPTCP cookie offering from server (seems OK).
3. MPTCP SYN+DATA+COOKIE from client (seems OK).
4. subsequent write + read on the opened socket (seems OK).
===Changes between v8 and v9
– Code is refactored (Max. reuse of existing linux kernel code).
– impact on fastopen.c is reduced to minima due to added function
subflow_v4_send_synack().
===Future work
-Adress the appearance of “MPTCP FIN” as duplicated acks.
-Integrate presented in the last patch selftests.

_PATCH v10_ client support. (Expand me)

Link to the mailing list of the linux upstream mptcp
Compared to the previous patches, these focus on the sender side.

Userspace programs:

Server specifics:
Create socket for listening in C with IPPROTO_MPTCP:

Set socket option “TCP_FASTOPEN” implemented by the patch.

Client specifics:
Set IPPROTO_MPTCP with MSG_FASTOPEN option on the client side.

MPTCP Fast Open in linux kernel

Related