MPTCP Fast Open in linux kernel

This set of patches will bring “Fast Open” Option support to MPTCP in linux kernel. The aim of Fast Open Mechanism is to eliminate one round trip time from a MPTCP conversation by allowing data to be included as part of the SYN segment that initiates the connection.

IETF RFC 8684: Appendix B. TCP Fast Open and MPTCP. IETF RFC LINK

These new patches (LWN | ML: mptcp.lists.linux.dev) provide the full support of TFO with MPTCP,
which means:

– MSG_FASTOPEN sendmsg flag support (patch 1/8)
– TFO support for the listener side (patches 2, 3, 4, 5/8)
– TCP_FASTOPEN socket option (patch 6/8)
– TCP_FASTOPEN_KEY socket option (patch 7/8)
– SELFTESTS (patch 8/8)

2 methods to Request a cookie (in MPTCP) are currently exist:

1) Listener side + Initator side with: MSG_FASTOPEN
LWN publication: https://lwn.net/Articles/916023/
2) Initiator side with: TCP_FASTOPEN_CONNECT
LWN publication https://lwn.net/Articles/909590/

Code examples for initiator side:
MSG_FASTOPEN (Updated on 6 Nov. 2022)
Here is example of Initiator in userspace program written in C:

PacketDrill test: Link to Github
PacketDrill test result: Link to Github

TCP_FASTOPEN_CONNECT

Here is example of Initiator in userspace program written in C:

Environment configuration (linux) for TCP_FASTOPEN:

Versions and their details are presented below:

_PATCH v2_ includes client-server partial support + more impact on existing TCP and MPTCP code. (Expand me)

1. MPTCP cookie request from client.
2. MPTCP cookie offering from server.
3. MPTCP SYN+DATA+COOKIE from client.
4. subsequent write + read on the opened socket.

This patch is Work In Progress and an early draft.

The patch is build against 5.16.0-rc8 net-next.

The patch is located in the next mailing list: https://patchew.org/MPTCP/[email protected]/

The pcap dump:

_PATCH v4_ client-server partial support + less impact on existing TCP and MPTCP code. (Expand me)

The differences between previous versions and v4:
1. An attempt to reduce impact on existing TCP code.
2. 2 files related to mptcp_fastopen are created(*.h + *.c).
3. “subflow_v4_conn_request” is used to call “mptcp_conn_request”(located in “mptcp_fastopen.c”) to process the received packet on the listener side when “SYN” is received during 3way handshake.
4. This chain adds “skb” to “&msk->sk_receive_queue” (“subflow_v4_conn_request”->”mptcp_conn_request”->”mptcp_try_fastopen”->”mptcp_fastopen_create_child”->”mptcp_fastopen_add_skb”)
5. Some minor comments from mailing list are not yet included in the current version of the PATCH.

The patch is located in the next mailing list: https://lore.kernel.org/mptcp/[email protected]/

You can download a TCPDUMP pcap from this link: Download tcpdump pcap

Thoughts on RST issue:
With gdb I got a backtrace for “tcp_send_active_reset”:

We can see here that it executed from “subflow_check_data_avail” (“/net/mptcp/subflow.c”)

We now check the “subflow_check_data_avail” line 1260 and in code snippet#2 see that it is located in the fallback “section” that leads us to possible lines, that lead us in code snippet#1 -> MAPPING_INVALID or MUPPING_DUMMY.

Further investigation of MAPPING INVALID seems to lead us to line 1081 of code snippet #3.

Backtrace:

code snippet#1

code snippet#2

code snippet #3

_PATCH v5_ client-server partial support + Multiple improvements. (Expand me)
Source code:
Link to the mailing list of the linux upstream mptcp
or
Link to the PATCHWORK of the linux upstream mptcp

===Changes between v4 and v5
-All data sent by the initiator received by listener in userspace.
-No more infinite re-transmissions of the same packet in the end of the second connection.
-”WARNING in sk stream kill queues”, when skb is added to msk queue – is fixed.
-Fix the second (discarded) packet that is re-transmitted in the end of 2nd session.
-We have (in the v4 it is not present) the FIN in the end of the 2nd (TFO data+cookie+syn) session.
===Tradeoffs of v5:
-Minimal impact on existing TCP code in linux kernel.
-Icreased size of the patch. Some functions are inspired from existing code in fastopen.c and others.
===Future work:
Address the appearance of ”MPTCP FIN” as duplicated acks.


_PATCH v6_ client-server partial support + Multiple improvements. (Expand me)
Source code:
Link to the mailing list of the linux upstream mptcp

===Changes between v5 and v6

– Change commit length
– Removed rmem commit (refactor)
– fixed setsocketopts
– separated helpers from add_skb_to_msk_fastopen into different patch.
===Tradeoffs of v5:
-Minimal impact on existing TCP code in linux kernel.
-Icreased size of the patc,
===Future work:
Address the appearance of ”MPTCP FIN” as duplicated acks.


_PATCH v7_ client-server partial support + Multiple improvements. (Expand me)

Source code:
Link to the mailing list of the linux upstream mptcp

[PATCH v7] includes “client-server” partial support for:
1. MPTCP cookie request from client (seems OK).
2. MPTCP cookie offering from server (seems OK).
3. MPTCP SYN+DATA+COOKIE from client (seems OK).
4. subsequent write + read on the opened socket (seems OK).

===Changes between v6 and v7
– Order of commits is changed.
– Code is refactored
===Pros/Cons of v7:
– little modifications of existing TCP code in linux kernel.
– relatively decreased size of the patch (in comparison with v6)
due to reuse of some functions in tcp_fastopen.c.
===Future work:
-Adress the appearance of “MPTCP FIN” as duplicated acks.

_PATCH v8_ client-server partial support + Refactored code (beginning). (Expand me)
Link to the mailing list of the linux upstream mptcp

[PATCH v8] includes “client-server” partial support for:
1. MPTCP cookie request from client (seems OK).
2. MPTCP cookie offering from server (seems OK).
3. MPTCP SYN+DATA+COOKIE from client (seems OK).
4. subsequent write + read on the opened socket (seems OK).
===Changes between v7 and v8
– We change from 0impact approach on existing TCP code to
full reusage of existing and available functions.
– Code is refactored (Max. reuse of existing linux kernel code).
– fastopen.c is reduced to minima.
– Other comments from mailing list are coming in the next version.
===Future work
-Adress the appearance of “MPTCP FIN” as duplicated acks.
-Integrate presented in the last patch selftests.

_PATCH v9_ client-server partial support + advanced refactoring. (Expand me)
Link to the mailing list of the linux upstream mptcp

[PATCH v9] includes “client-server” partial support for:
1. MPTCP cookie request from client (seems OK).
2. MPTCP cookie offering from server (seems OK).
3. MPTCP SYN+DATA+COOKIE from client (seems OK).
4. subsequent write + read on the opened socket (seems OK).
===Changes between v8 and v9
– Code is refactored (Max. reuse of existing linux kernel code).
– impact on fastopen.c is reduced to minima due to added function
subflow_v4_send_synack().
===Future work
-Adress the appearance of “MPTCP FIN” as duplicated acks.
-Integrate presented in the last patch selftests.


_PATCH v10_ client support. (Expand me)
Link to the mailing list of the linux upstream mptcp
Compared to the previous patches, these focus on the sender side.



Userspace programs:

Server specifics:
Create socket for listening in C with IPPROTO_MPTCP:

Set socket option “TCP_FASTOPEN” implemented by the patch.

Client specifics:
Set IPPROTO_MPTCP with MSG_FASTOPEN option on the client side.