there must be a bug. I get a lot of RSTs when browsing in a netns with a TUN driven by it

[ple1n]

 nsproxy[214777]: 2023-12-12T11:32:44.167596Z  INFO ipstack: NetworkPacket { ip: Version4(Ipv4Header { ihl: 5, differentiated_services_code_point: 0, explicit_congestion_notification: 0, payload_len: 477, identification: 0, dont_fragment: true, more_fragments: false, fragments_offset: 0, time_to_live: 64, protocol: 6, header_checksum: 0, source: [198, 18, 0, 0], destination: [100, 64, 0, 2], options: [] }, Ipv4Extensions { auth: None }), transport: Tcp(TcpHeader { source_port: 80, destination_port: 33816, sequence_number: 1251, acknowledgment_number: 1444113908, data_offset: 5, ns: false, fin: false, syn: false, rst: false, psh: true, ack: true, urg: false, ece: false, cwr: false, window_size: 16384, checksum: 27942, urgent_pointer: 0, options: [] }) }
 nsproxy[214777]: 2023-12-12T11:32:44.168245Z  INFO ipstack: NetworkPacket { ip: Version4(Ipv4Header { ihl: 5, differentiated_services_code_point: 0, explicit_congestion_notification: 0, payload_len: 20, identification: 0, dont_fragment: true, more_fragments: false, fragments_offset: 0, time_to_live: 64, protocol: 6, header_checksum: 0, source: [198, 18, 0, 0], destination: [100, 64, 0, 2], options: [] }, Ipv4Extensions { auth: None }), transport: Tcp(TcpHeader { source_port: 80, destination_port: 33816, sequence_number: 1708, acknowledgment_number: 1444113909, data_offset: 5, ns: false, fin: true, syn: false, rst: false, psh: false, ack: true, urg: false, ece: false, cwr: false, window_size: 16384, checksum: 63073, urgent_pointer: 0, options: [] }) }
 nsproxy[214777]: 2023-12-12T11:32:49.169809Z TRACE ipstack::stream::tcp: timeout reached for 198.18.0.0:80
 nsproxy[214777]: 2023-12-12T11:32:49.170011Z  INFO tun2socks5: Ending #1 TCP 100.64.0.2:33816 -> example.com.:80 with Err(Kind(TimedOut))
 nsproxy[214777]: 2023-12-12T11:32:49.170298Z TRACE tun2socks5: Session count 1

I'm trying to solve it. A whole TCP stack written by hand. Wcgw.

[ple1n]

check_pkt_type seems buggy. The last packet should be ACKing the FIN but it's invalidated

[ple1n]

I did a workaround. ple1n/ipstack@2dc5516

[SajjadPourali]

check_pkt_type seems buggy. The last packet should be ACKing the FIN but it's invalidated

I hope the problem is limited only to check_pkt_type because it would be easy to fix.

---

Could you please provide a pcap file and mention the OS that you use?

[ple1n]

heres the cap of curl example.com on linux 6.6.5-arch1-1 #1 SMP PREEMPT_DYNAMIC. https://github.com/planetoryd/misc/blob/master/cap_ipstack_rst.pcapng with https://github.com/ssrlive/ipstack/commit/df0765ab7636f7e9d2846b779bf412e15a205faa

with dbg!s https://github.com/planetoryd/misc/blob/master/ipstack_logs.md

the workarounded version https://github.com/planetoryd/misc/blob/master/ipstack_workaround.pcapng has RST+ACKs, which should be RSTing the connection and immediate restablishment (i guess) . on ple1n/ipstack@2dc5516

If you see large captures it's me using it as some kind of daily driver. therefore just ask if you want any data, or testing. and I use geph.io 's proxy as upstream, which is a tor-bridge like proxy service (better than tor bridge in fact)

It looks quite terrifying to have that much unverified logic. perhaps you can try https://github.com/verus-lang/verus but idk how long it will take to become usable. some nice formal verification (unlikely)

[SajjadPourali]

heres the cap of curl example.com on linux 6.6.5-arch1-1 #1 SMP PREEMPT_DYNAMIC. https://github.com/planetoryd/misc/blob/master/cap_ipstack_rst.pcapng with ssrlive@df0765a

with dbg!s https://github.com/planetoryd/misc/blob/master/ipstack_logs.md

the workarounded version https://github.com/planetoryd/misc/blob/master/ipstack_workaround.pcapng has RST+ACKs, which should be RSTing the connection and immediate restablishment (i guess) . on planetoryd@2dc5516

If you see large captures it's me using it as some kind of daily driver. therefore just ask if you want any data, or testing. and I use geph.io 's proxy as upstream, which is a tor-bridge like proxy service (better than tor bridge in fact)

It looks quite terrifying to have that much unverified logic. perhaps you can try https://github.com/verus-lang/verus but idk how long it will take to become usable. some nice formal verification (unlikely)

Fortunately, the logic is almost correct. There is only one minor issue that should not affect the communications, but it is indeed a bug! The IP stack deallocates the TCP stream after receiving the last FIN/ACK, so the final ACK goes unrecognized, which results in a RST being returned. In the correct implementation, it should handle the last ACK as well. However, in both cases—whether it returns an RST (the faulty behavior) or accepts the ACK without returning an RST — the connection will be closed.

Thanks for the report, I'll fix it soon. Did you see any other unexpected rst except the last ack for the first one?

---

I still did not checked your modifications on ple1n/ipstack@2dc5516 .

[SajjadPourali]

@planetoryd: Could you please verify that if #7 fixes the issue?

[ssrlive]

It's not fixed. @SajjadPourali

You can test ipstack with tun2socks5 under Linux. steps like this

sudo apt install iperf3 dante-server
sudo systemctl stop danted

git clone https://github.com/ssrlive/tun2socks5.git
cd tun2socks5

cargo b -r

# start testing
sudo scripts/iperf3.sh

# clear up
sudo sh -c "pkill tun2socks5; pkill iperf3; pkill danted; ip link del tun0; ip netns del test"

[ple1n]

Yeah, the following RST got removed, when I did the curl again.

I open my browser, mozilla connects something. It constantly times out, RSTs the connections and reconnect immediately.

https://github.com/planetoryd/misc/blob/master/timeout.pcapng

[ple1n]

2023-12-13T15:04:48.300632Z  INFO tun2socks5::dns: Allocate VirtDNS Name example.com.
2023-12-13T15:04:48.301102Z  INFO tun2socks5::dns: Allocate VirtDNS Name example.com.
2023-12-13T15:04:48.302699Z  INFO tun2socks5: Beginning #1 TCP 100.64.0.2:49738 -> example.com.:80
2023-12-13T15:06:18.691060Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:18.691528Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:18.695496Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:18.695741Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:18.697228Z  INFO tun2socks5: Beginning #4 TCP 100.64.0.2:39226 -> push.services.mozilla.com.:443
2023-12-13T15:06:19.448777Z  INFO tun2socks5::dns: Allocate VirtDNS Name r3.o.lencr.org.
2023-12-13T15:06:19.449152Z  INFO tun2socks5::dns: Allocate VirtDNS Name r3.o.lencr.org.
2023-12-13T15:06:19.450398Z  INFO tun2socks5: Beginning #6 TCP 100.64.0.2:48484 -> r3.o.lencr.org.:80
2023-12-13T15:06:20.047536Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:20.047886Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:20.049742Z  INFO tun2socks5: Beginning #8 TCP 100.64.0.2:39238 -> push.services.mozilla.com.:443
2023-12-13T15:06:20.971163Z  INFO tun2socks5: Beginning #9 UDP 100.64.0.2:49230 -> push.services.mozilla.com.:443
2023-12-13T15:06:21.274665Z ERROR tun2socks5: #9 UDP 100.64.0.2:49230 -> push.services.mozilla.com.:443 error "Invalid argument (os error 22)"
2023-12-13T15:06:24.841364Z  INFO tun2socks5: Ending #6 TCP 100.64.0.2:48484 -> r3.o.lencr.org.:80 with Err(Kind(TimedOut))
2023-12-13T15:06:25.047877Z  INFO tun2socks5: Ending #4 TCP 100.64.0.2:39226 -> push.services.mozilla.com.:443 with Err(Kind(TimedOut))
2023-12-13T15:06:26.389020Z  INFO tun2socks5: Ending #8 TCP 100.64.0.2:39238 -> push.services.mozilla.com.:443 with Err(Kind(TimedOut))
2023-12-13T15:06:31.394497Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:31.397211Z  INFO tun2socks5: Beginning #11 TCP 100.64.0.2:60064 -> push.services.mozilla.com.:443
2023-12-13T15:06:31.646661Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:31.648910Z  INFO tun2socks5: Beginning #13 TCP 100.64.0.2:60068 -> push.services.mozilla.com.:443
2023-12-13T15:06:32.132862Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:32.133311Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:32.135715Z  INFO tun2socks5: Beginning #15 TCP 100.64.0.2:60076 -> push.services.mozilla.com.:443
2023-12-13T15:06:37.133184Z  INFO tun2socks5: Ending #11 TCP 100.64.0.2:60064 -> push.services.mozilla.com.:443 with Err(Kind(TimedOut))

[SajjadPourali]

Yeah, the following RST got removed, when I did the curl again.

I open my browser, mozilla connects something. It constantly times out, RSTs the connections and reconnect immediately.

https://github.com/planetoryd/misc/blob/master/timeout.pcapng

The timeout issue is manageable because I have hard-coded a 5-second timeout (which I intend to make configurable #4). Firefox keeps the connections open for reuse, and in some cases, this can result in them remaining open for more than 30 seconds.

[SajjadPourali]

It's not fixed. @SajjadPourali

You can test ipstack with tun2socks5 under Linux. steps like this

sudo apt install iperf3 dante-server
sudo systemctl stop danted

git clone https://github.com/ssrlive/tun2socks5.git
cd tun2socks5

cargo b -r

# start testing
sudo scripts/iperf3.sh

# clear up
sudo sh -c "pkill tun2socks5; pkill iperf3; pkill danted; ip link del tun0; ip netns del test"

I'll set up a Linux machine over the weekend to test tun2socks5. In the meantime, if you could attach a .pcap file, I would be able to check it manually beforehand.

[SajjadPourali]

Yeah, the following RST got removed, when I did the curl again.

I open my browser, mozilla connects something. It constantly times out, RSTs the connections and reconnect immediately.

https://github.com/planetoryd/misc/blob/master/timeout.pcapng

I just changed the API to use a custom timeout and increased the TCP and UDP timeouts to 60 and 30, respectively, in the main branch.