You are here
Dropped Packets on MariaDB/MySQL Linux Servers
Table of Contents
ifconfig
ip
ethtool --statistics
nstat
netstat --statistics
ss
- Reasons for dropped packets
- Features of Network card
- Features of Network driver
- Network driver ring buffer
- TCP socket received and send buffer
- Flow Control
- Check over time:
sar
- Dropwatch
perf
tcpdump
/proc/net/netstat
- System Tap (
stap
) - Literature
When we do MariaDB and MySQL Database server analysis we see from time to time systems with a lot of dropped packets on a network interface:
ifconfig
# export DEV=enp0s25 # ifconfig ${DEV} eth0 Link encap:Ethernet HWaddr d0:50:99:70:f8:8e inet addr:192.168.1.135 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:4005096 errors:0 dropped:365956 overruns:0 frame:0 TX packets:3293230 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2897493579 (2.8 GB) TX bytes:461542567 (461.5 MB)
As we all learned ifconfig
is old and thus bad so we should use the new command ip
instead. See also our new hint: Replacement for ifconfig
statistics.
ip
# ip --statistics [--human-readable] address show ${DEV} 2: eth0:mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether d0:50:99:70:f8:8e brd ff:ff:ff:ff:ff:ff inet 192.168.1.135/24 brd 192.168.1.255 scope global eth0 valid_lft forever preferred_lft forever RX: bytes packets errors dropped overrun mcast 2897493579 4005096 0 365956 0 19703 TX: bytes packets errors dropped carrier collsns 461542567 3293230 0 0 0 0
ethtool --statistics
If you want to see some more statistics about your packets you can run:
# ethtool --statistics ${DEV} NIC statistics: rx_packets: 4134 tx_packets: 3507 rx_bytes: 2791271 tx_bytes: 607408 rx_broadcast: 59 tx_broadcast: 82 rx_multicast: 89 tx_multicast: 85 rx_errors: 0 tx_errors: 0 tx_dropped: 0 multicast: 89 collisions: 0 rx_length_errors: 0 rx_over_errors: 0 rx_crc_errors: 0 rx_frame_errors: 0 rx_no_buffer_count: 0 rx_missed_errors: 0 tx_aborted_errors: 0 tx_carrier_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 tx_window_errors: 0 tx_abort_late_coll: 0 tx_deferred_ok: 0 tx_single_coll_ok: 0 tx_multi_coll_ok: 0 tx_timeout_count: 0 tx_restart_queue: 0 rx_long_length_errors: 0 rx_short_length_errors: 0 rx_align_errors: 0 tx_tcp_seg_good: 53 tx_tcp_seg_failed: 0 rx_flow_control_xon: 0 rx_flow_control_xoff: 0 tx_flow_control_xon: 0 tx_flow_control_xoff: 0 rx_csum_offload_good: 3983 rx_csum_offload_errors: 0 rx_header_split: 0 alloc_rx_buff_failed: 0 tx_smbus: 0 rx_smbus: 53 dropped_smbus: 0 rx_dma_failed: 0 tx_dma_failed: 0 rx_hwtstamp_cleared: 0 uncorr_ecc_errors: 0 corr_ecc_errors: 0 tx_hwtstamp_timeouts: 0 tx_hwtstamp_skipped: 0
But unfortunately RX dropped is not listed but only TX dropped.
nstat
Detailed network statistics should be checked with nstat
today:
# nstat --ignore --noupdate --zero #kernel SctpCurrEstab 0 0.0 SctpActiveEstabs 0 0.0 SctpPassiveEstabs 0 0.0 SctpAborteds 0 0.0 SctpShutdowns 0 0.0 SctpOutOfBlues 0 0.0 SctpChecksumErrors 0 0.0 SctpOutCtrlChunks 0 0.0 SctpOutOrderChunks 0 0.0 SctpOutUnorderChunks 0 0.0 SctpInCtrlChunks 0 0.0 SctpInOrderChunks 0 0.0 SctpInUnorderChunks 0 0.0 SctpFragUsrMsgs 0 0.0 SctpReasmUsrMsgs 0 0.0 SctpOutSCTPPacks 0 0.0 SctpInSCTPPacks 0 0.0 SctpT1InitExpireds 0 0.0 SctpT1CookieExpireds 0 0.0 SctpT2ShutdownExpireds 0 0.0 SctpT3RtxExpireds 0 0.0 SctpT4RtoExpireds 0 0.0 SctpT5ShutdownGuardExpireds 0 0.0 SctpDelaySackExpireds 0 0.0 SctpAutocloseExpireds 0 0.0 SctpT3Retransmits 0 0.0 SctpPmtudRetransmits 0 0.0 SctpFastRetransmits 0 0.0 SctpInPktSoftirq 0 0.0 SctpInPktBacklog 0 0.0 SctpInPktDiscards 0 0.0 SctpInDataChunkDiscards 0 0.0 IpInReceives 2146596 0.0 IpInHdrErrors 0 0.0 IpInAddrErrors 2 0.0 IpForwDatagrams 0 0.0 IpInUnknownProtos 0 0.0 IpInDiscards 0 0.0 IpInDelivers 2146589 0.0 IpOutRequests 1821062 0.0 IpOutDiscards 20 0.0 IpOutNoRoutes 3 0.0 IpReasmTimeout 0 0.0 IpReasmReqds 6 0.0 IpReasmOKs 3 0.0 IpReasmFails 0 0.0 IpFragOKs 0 0.0 IpFragFails 0 0.0 IpFragCreates 0 0.0 IcmpInMsgs 7434 0.0 IcmpInErrors 14 0.0 IcmpInCsumErrors 0 0.0 IcmpInDestUnreachs 3703 0.0 IcmpInTimeExcds 0 0.0 IcmpInParmProbs 0 0.0 IcmpInSrcQuenchs 0 0.0 IcmpInRedirects 0 0.0 IcmpInEchos 610 0.0 IcmpInEchoReps 3121 0.0 IcmpInTimestamps 0 0.0 IcmpInTimestampReps 0 0.0 IcmpInAddrMasks 0 0.0 IcmpInAddrMaskReps 0 0.0 IcmpOutMsgs 12373 0.0 IcmpOutErrors 0 0.0 IcmpOutDestUnreachs 3701 0.0 IcmpOutTimeExcds 0 0.0 IcmpOutParmProbs 0 0.0 IcmpOutSrcQuenchs 0 0.0 IcmpOutRedirects 0 0.0 IcmpOutEchos 8062 0.0 IcmpOutEchoReps 610 0.0 IcmpOutTimestamps 0 0.0 IcmpOutTimestampReps 0 0.0 IcmpOutAddrMasks 0 0.0 IcmpOutAddrMaskReps 0 0.0 IcmpMsgInType0 3121 0.0 IcmpMsgInType3 3703 0.0 IcmpMsgInType8 610 0.0 IcmpMsgOutType0 610 0.0 IcmpMsgOutType3 3701 0.0 IcmpMsgOutType8 8062 0.0 TcpActiveOpens 31370 0.0 TcpPassiveOpens 18080 0.0 TcpAttemptFails 2997 0.0 TcpEstabResets 56 0.0 TcpInSegs 2166708 0.0 TcpOutSegs 1842768 0.0 TcpRetransSegs 4093 0.0 TcpInErrs 5 0.0 TcpOutRsts 2905 0.0 TcpInCsumErrors 0 0.0 UdpInDatagrams 40549 0.0 UdpNoPorts 41 0.0 UdpInErrors 0 0.0 UdpOutDatagrams 38161 0.0 UdpRcvbufErrors 0 0.0 UdpSndbufErrors 0 0.0 UdpInCsumErrors 0 0.0 UdpIgnoredMulti 181 0.0 UdpLiteInDatagrams 0 0.0 UdpLiteNoPorts 0 0.0 UdpLiteInErrors 0 0.0 UdpLiteOutDatagrams 0 0.0 UdpLiteRcvbufErrors 0 0.0 UdpLiteSndbufErrors 0 0.0 UdpLiteInCsumErrors 0 0.0 UdpLiteIgnoredMulti 0 0.0 Ip6InReceives 71921 0.0 Ip6InHdrErrors 0 0.0 Ip6InTooBigErrors 0 0.0 Ip6InNoRoutes 1 0.0 Ip6InAddrErrors 0 0.0 Ip6InUnknownProtos 0 0.0 Ip6InTruncatedPkts 0 0.0 Ip6InDiscards 0 0.0 Ip6InDelivers 71920 0.0 Ip6OutForwDatagrams 0 0.0 Ip6OutRequests 68903 0.0 Ip6OutDiscards 0 0.0 Ip6OutNoRoutes 47 0.0 Ip6ReasmTimeout 0 0.0 Ip6ReasmReqds 0 0.0 Ip6ReasmOKs 0 0.0 Ip6ReasmFails 0 0.0 Ip6FragOKs 0 0.0 Ip6FragFails 0 0.0 Ip6FragCreates 0 0.0 Ip6InMcastPkts 2060 0.0 Ip6OutMcastPkts 94 0.0 Ip6InOctets 80749746 0.0 Ip6OutOctets 9844802 0.0 Ip6InMcastOctets 247993 0.0 Ip6OutMcastOctets 12725 0.0 Ip6InBcastOctets 0 0.0 Ip6OutBcastOctets 0 0.0 Ip6InNoECTPkts 103468 0.0 Ip6InECT1Pkts 1 0.0 Ip6InECT0Pkts 0 0.0 Ip6InCEPkts 1 0.0 Icmp6InMsgs 3054 0.0 Icmp6InErrors 0 0.0 Icmp6OutMsgs 1948 0.0 Icmp6OutErrors 0 0.0 Icmp6InCsumErrors 0 0.0 Icmp6InDestUnreachs 0 0.0 Icmp6InPktTooBigs 0 0.0 Icmp6InTimeExcds 0 0.0 Icmp6InParmProblems 0 0.0 Icmp6InEchos 776 0.0 Icmp6InEchoReplies 0 0.0 Icmp6InGroupMembQueries 294 0.0 Icmp6InGroupMembResponses 0 0.0 Icmp6InGroupMembReductions 0 0.0 Icmp6InRouterSolicits 0 0.0 Icmp6InRouterAdvertisements 836 0.0 Icmp6InNeighborSolicits 684 0.0 Icmp6InNeighborAdvertisements 462 0.0 Icmp6InRedirects 0 0.0 Icmp6InMLDv2Reports 2 0.0 Icmp6OutDestUnreachs 0 0.0 Icmp6OutPktTooBigs 0 0.0 Icmp6OutTimeExcds 0 0.0 Icmp6OutParmProblems 0 0.0 Icmp6OutEchos 0 0.0 Icmp6OutEchoReplies 776 0.0 Icmp6OutGroupMembQueries 0 0.0 Icmp6OutGroupMembResponses 0 0.0 Icmp6OutGroupMembReductions 0 0.0 Icmp6OutRouterSolicits 1 0.0 Icmp6OutRouterAdvertisements 0 0.0 Icmp6OutNeighborSolicits 464 0.0 Icmp6OutNeighborAdvertisements 684 0.0 Icmp6OutRedirects 0 0.0 Icmp6OutMLDv2Reports 23 0.0 Icmp6InType128 776 0.0 Icmp6InType130 294 0.0 Icmp6InType134 836 0.0 Icmp6InType135 684 0.0 Icmp6InType136 462 0.0 Icmp6InType143 2 0.0 Icmp6OutType129 776 0.0 Icmp6OutType133 1 0.0 Icmp6OutType135 464 0.0 Icmp6OutType136 684 0.0 Icmp6OutType143 23 0.0 Udp6InDatagrams 248 0.0 Udp6NoPorts 0 0.0 Udp6InErrors 0 0.0 Udp6OutDatagrams 153 0.0 Udp6RcvbufErrors 0 0.0 Udp6SndbufErrors 0 0.0 Udp6InCsumErrors 0 0.0 Udp6IgnoredMulti 0 0.0 UdpLite6InDatagrams 0 0.0 UdpLite6NoPorts 0 0.0 UdpLite6InErrors 0 0.0 UdpLite6OutDatagrams 0 0.0 UdpLite6RcvbufErrors 0 0.0 UdpLite6SndbufErrors 0 0.0 UdpLite6InCsumErrors 0 0.0 TcpExtSyncookiesSent 0 0.0 TcpExtSyncookiesRecv 0 0.0 TcpExtSyncookiesFailed 0 0.0 TcpExtEmbryonicRsts 0 0.0 TcpExtPruneCalled 0 0.0 TcpExtRcvPruned 0 0.0 TcpExtOfoPruned 0 0.0 TcpExtOutOfWindowIcmps 0 0.0 TcpExtLockDroppedIcmps 0 0.0 TcpExtArpFilter 0 0.0 TcpExtTW 20345 0.0 TcpExtTWRecycled 0 0.0 TcpExtTWKilled 0 0.0 TcpExtPAWSActive 0 0.0 TcpExtPAWSEstab 2 0.0 TcpExtDelayedACKs 15602 0.0 TcpExtDelayedACKLocked 9 0.0 TcpExtDelayedACKLost 114 0.0 TcpExtListenOverflows 0 0.0 TcpExtListenDrops 0 0.0 TcpExtTCPHPHits 1591674 0.0 TcpExtTCPPureAcks 96024 0.0 TcpExtTCPHPAcks 92591 0.0 TcpExtTCPRenoRecovery 0 0.0 TcpExtTCPSackRecovery 3 0.0 TcpExtTCPSACKReneging 0 0.0 TcpExtTCPSACKReorder 84 0.0 TcpExtTCPRenoReorder 0 0.0 TcpExtTCPTSReorder 0 0.0 TcpExtTCPFullUndo 0 0.0 TcpExtTCPPartialUndo 0 0.0 TcpExtTCPDSACKUndo 0 0.0 TcpExtTCPLossUndo 7 0.0 TcpExtTCPLostRetransmit 1 0.0 TcpExtTCPRenoFailures 0 0.0 TcpExtTCPSackFailures 0 0.0 TcpExtTCPLossFailures 0 0.0 TcpExtTCPFastRetrans 31 0.0 TcpExtTCPSlowStartRetrans 0 0.0 TcpExtTCPTimeouts 3774 0.0 TcpExtTCPLossProbes 69 0.0 TcpExtTCPLossProbeRecovery 3 0.0 TcpExtTCPRenoRecoveryFail 0 0.0 TcpExtTCPSackRecoveryFail 0 0.0 TcpExtTCPRcvCollapsed 0 0.0 TcpExtTCPDSACKOldSent 109 0.0 TcpExtTCPDSACKOfoSent 1 0.0 TcpExtTCPDSACKRecv 25 0.0 TcpExtTCPDSACKOfoRecv 0 0.0 TcpExtTCPAbortOnData 548 0.0 TcpExtTCPAbortOnClose 29 0.0 TcpExtTCPAbortOnMemory 0 0.0 TcpExtTCPAbortOnTimeout 27 0.0 TcpExtTCPAbortOnLinger 0 0.0 TcpExtTCPAbortFailed 0 0.0 TcpExtTCPMemoryPressures 0 0.0 TcpExtTCPMemoryPressuresChrono 0 0.0 TcpExtTCPSACKDiscard 0 0.0 TcpExtTCPDSACKIgnoredOld 0 0.0 TcpExtTCPDSACKIgnoredNoUndo 18 0.0 TcpExtTCPSpuriousRTOs 0 0.0 TcpExtTCPMD5NotFound 0 0.0 TcpExtTCPMD5Unexpected 0 0.0 TcpExtTCPMD5Failure 0 0.0 TcpExtTCPSackShifted 28 0.0 TcpExtTCPSackMerged 20 0.0 TcpExtTCPSackShiftFallback 95 0.0 TcpExtTCPBacklogDrop 0 0.0 TcpExtPFMemallocDrop 0 0.0 TcpExtTCPMinTTLDrop 0 0.0 TcpExtTCPDeferAcceptDrop 1 0.0 TcpExtIPReversePathFilter 0 0.0 TcpExtTCPTimeWaitOverflow 0 0.0 TcpExtTCPReqQFullDoCookies 0 0.0 TcpExtTCPReqQFullDrop 0 0.0 TcpExtTCPRetransFail 0 0.0 TcpExtTCPRcvCoalesce 200321 0.0 TcpExtTCPOFOQueue 61316 0.0 TcpExtTCPOFODrop 0 0.0 TcpExtTCPOFOMerge 2 0.0 TcpExtTCPChallengeACK 5 0.0 TcpExtTCPSYNChallenge 5 0.0 TcpExtTCPFastOpenActive 0 0.0 TcpExtTCPFastOpenActiveFail 0 0.0 TcpExtTCPFastOpenPassive 0 0.0 TcpExtTCPFastOpenPassiveFail 0 0.0 TcpExtTCPFastOpenListenOverflow 0 0.0 TcpExtTCPFastOpenCookieReqd 0 0.0 TcpExtTCPFastOpenBlackhole 0 0.0 TcpExtTCPSpuriousRtxHostQueues 12 0.0 TcpExtBusyPollRxPackets 0 0.0 TcpExtTCPAutoCorking 4012 0.0 TcpExtTCPFromZeroWindowAdv 0 0.0 TcpExtTCPToZeroWindowAdv 0 0.0 TcpExtTCPWantZeroWindowAdv 188 0.0 TcpExtTCPSynRetrans 3737 0.0 TcpExtTCPOrigDataSent 437079 0.0 TcpExtTCPHystartTrainDetect 35 0.0 TcpExtTCPHystartTrainCwnd 1153 0.0 TcpExtTCPHystartDelayDetect 0 0.0 TcpExtTCPHystartDelayCwnd 0 0.0 TcpExtTCPACKSkippedSynRecv 0 0.0 TcpExtTCPACKSkippedPAWS 0 0.0 TcpExtTCPACKSkippedSeq 10 0.0 TcpExtTCPACKSkippedFinWait2 0 0.0 TcpExtTCPACKSkippedTimeWait 0 0.0 TcpExtTCPACKSkippedChallenge 0 0.0 TcpExtTCPWinProbe 0 0.0 TcpExtTCPKeepAlive 5134 0.0 TcpExtTCPMTUPFail 0 0.0 TcpExtTCPMTUPSuccess 0 0.0 TcpExtTCPWqueueTooBig 0 0.0 IpExtInNoRoutes 0 0.0 IpExtInTruncatedPkts 0 0.0 IpExtInMcastPkts 2893 0.0 IpExtOutMcastPkts 466 0.0 IpExtInBcastPkts 190 0.0 IpExtOutBcastPkts 15 0.0 IpExtInOctets 4837352803 0.0 IpExtOutOctets 1199138713 0.0 IpExtInMcastOctets 2244078 0.0 IpExtOutMcastOctets 40977 0.0 IpExtInBcastOctets 45209 0.0 IpExtOutBcastOctets 4035 0.0 IpExtInCsumErrors 0 0.0 IpExtInNoECTPkts 3140592 0.0 IpExtInECT1Pkts 0 0.0 IpExtInECT0Pkts 0 0.0 IpExtInCEPkts 0 0.0 IpExtReasmOverlaps 0 0.0
netstat --statistics
Old command netstat
but still widely used an known (use the previous nstat
instead):
# netstat --statistics ${DEV} TcpExt: 65 invalid SYN cookies received 155 resets received for embryonic SYN_RECV sockets 89 packets pruned from receive queue because of socket buffer overrun ArpFilter: 0 14341 TCP sockets finished time wait in fast timer 10 time wait sockets recycled by time stamp 713178 delayed acks sent 316 delayed acks further delayed because of locked socket Quick ack mode was activated 3908 times 29091413 packets directly queued to recvmsg prequeue. 19881778 packets directly received from backlog 1892094685 packets directly received from prequeue 31118010 packets header predicted 1272468 packets header predicted and directly queued to user TCPPureAcks: 1118011 TCPHPAcks: 30188520 TCPRenoRecovery: 2524 TCPSackRecovery: 1 TCPSACKReneging: 0 TCPFACKReorder: 0 TCPSACKReorder: 0 TCPRenoReorder: 277 TCPTSReorder: 128 TCPFullUndo: 1038 TCPPartialUndo: 640 TCPDSACKUndo: 0 TCPLossUndo: 1 TCPLoss: 0 TCPLostRetransmit: 0 TCPRenoFailures: 117 TCPSackFailures: 0 TCPLossFailures: 42 TCPFastRetrans: 1991 TCPForwardRetrans: 0 TCPSlowStartRetrans: 668 TCPTimeouts: 3276 TCPRenoRecoveryFail: 674 TCPSackRecoveryFail: 0 TCPSchedulerFailed: 2 TCPRcvCollapsed: 5439 TCPDSACKOldSent: 4 TCPDSACKOfoSent: 0 TCPDSACKRecv: 2 TCPDSACKOfoRecv: 0 TCPAbortOnSyn: 0 TCPAbortOnData: 14 TCPAbortOnClose: 56 TCPAbortOnMemory: 0 TCPAbortOnTimeout: 32 TCPAbortOnLinger: 0 TCPAbortFailed: 0 TCPMemoryPressures: 0
ss
Current replacement of netstat
is ss
:
# ss --summary Total: 1449 (kernel 3546) TCP: 79 (estab 19, closed 33, orphaned 0, synrecv 0, timewait 31/0), ports 0 Transport Total IP IPv6 * 3546 - - RAW 0 0 0 UDP 9 6 3 TCP 46 16 30 INET 55 22 33 FRAG 0 0 0
Reasons for dropped packets
- NIC ring buffers getting full and unable to cope-up with incoming bursts of traffic
- CPU receiving NIC interrupts is very busy and unable to process
- Some cable/hardware/duplex issues
- Some bug in NIC driver
- Softnet backlog full (Measured from
/proc/net/softnet_stat
) - Bad / Unintended VLAN tags
- Unknown / Unregistered protocols
- IPv6 frames when the server is not configured for IPv6
- Possibly some miss-configuration in Hypervisor (VMware)?
- Missing flow control?
Features of Network card
# ethtool ${DEV} Settings for enp0s25: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: No Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: No Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on MDI-X: on (auto) Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000007 (7) drv probe link Link detected: yes
Features of Network driver
# ethtool --show-features ${DEV} Features for enp0s25: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: off [fixed] tx-tcp-mangleid-segmentation: off tx-tcp6-segmentation: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off [fixed] receive-hashing: on highdma: on [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-gre-csum-segmentation: off [fixed] tx-ipxip4-segmentation: off [fixed] tx-ipxip6-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] tx-udp_tnl-csum-segmentation: off [fixed] tx-gso-partial: off [fixed] tx-sctp-segmentation: off [fixed] tx-esp-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off rx-all: off tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] hw-tc-offload: off [fixed] esp-hw-offload: off [fixed] esp-tx-csum-hw-offload: off [fixed] rx-udp_tunnel-port-offload: off [fixed]
Network driver ring buffer
# ethtool --show-ring ${DEV} Ring parameters for enp0s25: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 256 RX Mini: 0 RX Jumbo: 0 TX: 256 # ethtool --set-ring ${DEV} rx 4096 # ethtool --show-ring ${DEV} Ring parameters for enp0s25: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 256
This did not help in my case.
TCP socket received and send buffer
# /sbin/sysctl -a --pattern 'net.ipv4' | grep mem net.ipv4.igmp_max_memberships = 20 net.ipv4.tcp_mem = 186753 249007 373506 net.ipv4.tcp_rmem = 4096 87380 6291456 net.ipv4.tcp_wmem = 4096 16384 4194304 net.ipv4.udp_mem = 373509 498014 747018 net.ipv4.udp_rmem_min = 4096 net.ipv4.udp_wmem_min = 4096
Flow Control
If flow control is activated can be checked as follows:
# ethtool -a ${DEV} Pause parameters for enp0s25: Autonegotiate: on RX: on TX: on
Check over time: sar
To check the behaviour over time sar
is a good tool:
# sar -n EDEV 1 | grep -v lo Linux 4.15.0-70-generic (chef) 28/11/19 _x86_64_ (4 CPU) 11:56:51 IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s 11:56:52 enp0s25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:56:53 enp0s25 0.00 0.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 11:56:54 enp0s25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:56:55 enp0s25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Dropwatch
# git clone https://github.com/pavel-odintsov/drop_watch # cd drop_watch/src # make gcc -I/usr/include/libnl3/ -c -g -D_GNU_SOURCE -Wall -Werror main.c gcc -I/usr/include/libnl3/ -c -g -D_GNU_SOURCE -Wall -Werror lookup.c gcc -I/usr/include/libnl3/ -c -g -D_GNU_SOURCE -Wall -Werror lookup_bfd.c gcc -I/usr/include/libnl3/ -c -g -D_GNU_SOURCE -Wall -Werror lookup_kas.c gcc -I/usr/include/libnl3/ -g -o dropwatch main.o lookup.o lookup_bfd.o lookup_kas.o -lbfd -lreadline -lnl-3 -lnl-genl-3
# ./dropwatch -l kas Initalizing kallsyms db dropwatch> start Enabling monitoring... Kernel monitoring activated. Issue Ctrl-C to stop monitoring 4 drops at sk_stream_kill_queues+4d (0xffffffff8c05c22d) 1 drops at inet_sock_destruct+4d (0xffffffff8c0f9e8d) 2 drops at arp_error_report+41 (0xffffffff8c0f00f1) 4 drops at unix_stream_connect+2e5 (0xffffffff8c12bbf5) 1 drops at neigh_probe+61 (0xffffffff8c076e61) 1 drops at sk_stream_kill_queues+4d (0xffffffff8c05c22d) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) 2 drops at neigh_probe+61 (0xffffffff8c076e61) 4 drops at unix_stream_connect+2e5 (0xffffffff8c12bbf5) 3 drops at sk_stream_kill_queues+4d (0xffffffff8c05c22d) 2 drops at arp_error_report+41 (0xffffffff8c0f00f1) 1 drops at icmpv6_rcv+1c1 (0xffffffff8c159081) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) 2 drops at __netif_receive_skb_core+6b8 (0xffffffff8c06a498) 2 drops at __netif_receive_skb_core+6b8 (0xffffffff8c06a498) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) 4 drops at icmpv6_rcv+1c1 (0xffffffff8c159081) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) Got a stop message dropwatch> Terminating dropwatch...
Some probing led to the function: __netif_receive_skb_core+6b8
. But no clue what it is.
perf
perf record -g -a -e skb:kfree_skb perf script ... swapper 0 [003] 3161.113873: skb:kfree_skb: skbaddr=0xffff9247df562000 protocol=34939 location=0xffffffffb586a498 7fffb5856ec6 kfree_skb ([kernel.kallsyms]) 7fffb586c498 __netif_receive_skb_core ([kernel.kallsyms]) 7fffb586c998 __netif_receive_skb ([kernel.kallsyms]) 7fffb586ed25 netif_receive_skb_internal ([kernel.kallsyms]) 7fffb586faa5 napi_gro_receive ([kernel.kallsyms]) 7fffc01fa9c6 e1000_receive_skb ([kernel.kallsyms]) 7fffc01fc58e e1000_clean_rx_irq ([kernel.kallsyms]) 7fffc0203fee e1000e_poll ([kernel.kallsyms]) 7fffb586f180 net_rx_action ([kernel.kallsyms]) 7fffb5c020e4 __softirqentry_text_start ([kernel.kallsyms]) 7fffb5098bd5 irq_exit ([kernel.kallsyms]) 7fffb5a04736 do_IRQ ([kernel.kallsyms]) 7fffb5a0290c ret_from_intr ([kernel.kallsyms]) 7fffb5806337 cpuidle_enter_state ([kernel.kallsyms]) 7fffb58065b7 cpuidle_enter ([kernel.kallsyms]) 7fffb50d96c3 call_cpuidle ([kernel.kallsyms]) 7fffb50d993c do_idle ([kernel.kallsyms]) 7fffb50d9b83 cpu_startup_entry ([kernel.kallsyms]) 7fffb505c61b start_secondary ([kernel.kallsyms]) 7fffb50020d5 secondary_startup_64 ([kernel.kallsyms]) ...
tcpdump
# tcpdump --interface=${DEV} -w trace.pcap tcpdump: listening on enp0s25, link-type EN10MB (Ethernet), capture size 262144 bytes 642 packets captured 642 packets received by filter 0 packets dropped by kernel 5 packets dropped by interface
# tcpdump -n -t -i ${DEV} tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp0s25, link-type EN10MB (Ethernet), capture size 262144 bytes STP 802.1d, Config, Flags [none], bridge-id 8000.cc:5d:4e:2e:37:94.8002, length 47 IP chef > static.211.20.9.176.clients.your-server.de: ICMP echo request, id 7432, seq 1, length 64 IP chef.45753 > internetbox.home.domain: 51306+ [1au] PTR? 211.20.9.176.in-addr.arpa. (54) IP static.211.20.9.176.clients.your-server.de > chef: ICMP echo reply, id 7432, seq 1, length 64 IP chef.51876 > static.211.20.9.176.clients.your-server.de.2222: Flags [S], seq 2727621364, win 29200, options [mss 1460,sackOK,TS val 1819633171 ecr 0,nop,wscale 7], length 0 IP internetbox.home.domain > chef.45753: 51306 1/0/1 PTR static.211.20.9.176.clients.your-server.de. (110) IP chef.58204 > internetbox.home.domain: 37314+ [1au] PTR? 1.1.168.192.in-addr.arpa. (53) IP internetbox.home.domain > chef.58204: 37314* 1/0/1 PTR internetbox.home. (83) IP static.211.20.9.176.clients.your-server.de.2222 > chef.51876: Flags [R.], seq 0, ack 2727621365, win 0, length 0 IP chef > static.173.10.47.78.clients.your-server.de: ICMP echo request, id 7436, seq 1, length 64 IP chef.40629 > internetbox.home.domain: 32783+ [1au] PTR? 173.10.47.78.in-addr.arpa. (54) IP static.173.10.47.78.clients.your-server.de > chef: ICMP echo reply, id 7436, seq 1, length 64 IP chef.34262 > static.173.10.47.78.clients.your-server.de.2222: Flags [S], seq 2923371146, win 29200, options [mss 1460,sackOK,TS val 1741234319 ecr 0,nop,wscale 7], length 0 IP internetbox.home.domain > chef.40629: 32783 1/0/1 PTR static.173.10.47.78.clients.your-server.de. (110) e8:d1:1b:ec:98:30 (oui Unknown) > Broadcast, ethertype Unknown (0x88e1), length 60: 0x0000: 0120 6000 0000 0000 0000 0000 0000 0000 ..`............. 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. e8:d1:1b:ec:98:31 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: 0x0000: 0120 6000 0000 0000 0000 0000 0000 0000 ..`............. 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. IP chef.34262 > static.173.10.47.78.clients.your-server.de.2222: Flags [S], seq 2923371146, win 29200, options [mss 1460,sackOK,TS val 1741235343 ecr 0,nop,wscale 7], length 0 IP internetbox.home.mdns > 224.0.0.251.mdns: 0 [25a] [13q] PTR (QM)? _services._dns-sd._udp.local. PTR (QM)? _printer._tcp.local. PTR (QM)? _pdl-datastream._tcp.local. PTR (QM)? _ipp._tcp.local. PTR (QM)? _http._tcp.local. PTR (QM)? _scanner._tcp.local. PTR (QM)? _http-alt._tcp.local. PTR (QM)? _uscan._tcp.local. PTR (QM)? _ipps._tcp.local. PTR (QM)? _privet._tcp.local. PTR (QM)? _uscans._tcp.local. PTR (QM)? _mqtt._tcp.local. PTR (QM)? _ssw._tcp.local. (968) IP chef.43103 > internetbox.home.domain: 55791+ [1au] PTR? 251.0.0.224.in-addr.arpa. (53) IP internetbox.home.domain > chef.43103: 55791 NXDomain 0/1/1 (110) IP chef.43103 > internetbox.home.domain: 55791+ PTR? 251.0.0.224.in-addr.arpa. (42) IP internetbox.home.domain > chef.43103: 55791 NXDomain 0/0/0 (42) STP 802.1d, Config, Flags [none], bridge-id 8000.cc:5d:4e:2e:37:94.8002, length 47 23 packets captured 23 packets received by filter 0 packets dropped by kernel 1 packet dropped by interface
/proc/net/netstat
# ./myenv/utl/netstat.php --skip-zeroj | grep -i drop LockDroppedIcmps 0 ListenDrops 0 TCPBacklogDrop 0 PFMemallocDrop 0 TCPMinTTLDrop 0 TCPDeferAcceptDrop 0 TCPReqQFullDrop 0 TCPOFODrop 0
System Tap (stap
)
# stap --all-modules dropwatch.stp Monitoring for dropped packets 18 packets dropped at skb_queue_purge 8 packets dropped at unix_stream_connect 5 packets dropped at sk_stream_kill_queues 2 packets dropped at icmp_rcv 2 packets dropped at __netif_receive_skb_core 1 packets dropped at tcp_v4_do_rcv 1 packets dropped at tcp_v4_rcv Stopping dropped packet monitor
# stap --all-modules drop.stp Timestamp Src IP Dst IP IP Proto TCP Src Port TCP Dst Port Execname Wed Nov 27 21:55:51 2019 0.0.0.0 0.128.239.209 0 0 0 swapper/3 Wed Nov 27 21:55:51 2019 0.0.0.0 0.128.239.209 0 0 0 swapper/3 Wed Nov 27 21:56:06 2019 0.0.0.0 0.128.239.209 0 0 0 swapper/3 Wed Nov 27 21:56:06 2019 0.0.0.0 0.128.239.209 0 0 0 swapper/3 Wed Nov 27 21:56:36 2019 0.0.0.0 0.128.239.209 0 0 0 swapper/3 Wed Nov 27 21:56:36 2019 0.0.0.0 0.128.239.209 0 0 0 swapper/3
#! /usr/bin/env stap # # drop.stp # probe begin { printf("Timestamp Src IP Dst IP IP Proto TCP Src Port TCP Dst Port Execname\n") } probe end { printf("Stopping dropped packet monitor\n") } probe kernel.trace("kfree_skb") { if (symname($location) == "__netif_receive_skb_core") { printf("%s %s %s %5d %5d %5d %s\n", ctime(gettimeofday_s()), ip_ntop(__ip_skb_saddr($skb)), ip_ntop(__ip_skb_daddr($skb)), __ip_skb_proto($skb), __tcp_skb_dport($skb), __tcp_skb_sport($skb), execname() ) } }
If interpretation of IP protocol is correctly:
0x00 | 0 | HOPOPT | IPv6 | Hop-by-Hop Option | RFC 8200 |
No clue, what it finally is. Giving up...
Literature
- What exactly is an ifconfig dropped RX packet?
- ifconfig reports packet drop
Beginning with kernel 2.6.37, it has been changed the meaning of dropped packet count. Before, dropped packets was most likely due to an error. Now, the rx_dropped counter shows statistics for dropped frames because of:
- Softnet backlog full
- Bad / Unintended VLAN tags
- Unknown / Unregistered protocols
- IPv6 frames when the server is not configured for IPv6
- Finding out if/why a server is dropping packets
- ifconfig shows dropped rx packets
- Using netstat and dropwatch to observe packet loss on Linux servers
- Monitoring and Tuning the Linux Networking Stack: Receiving Data
- Monitoring and Tuning the Linux Networking Stack: Sending Data
- vSphere 6 experiencing high packet loss
- Network Troubleshooting Guide
dropwatch
- Linux man page- Brendan Gregg:
perf
Examples - Manpage of
tcpdump
- List of IP protocol numbers
- Monitoring Network Packets Drops in Kernel
- SystemTap
Comments
WMware snapshots or Veeam backups
Good explanation for bond interfaces
802.3ad bond interface have show high RX dropped packets
I found a god explanation for dropped packages on bonded interfaces: [ 1 ]
This is related to the bonding mode and _not_ a bug. The bonding module will drop duplicate frames received on inactive ports, which is normal behaviour. Overall the packets should be getting into the machine without problems since they are received on the active slave. To confirm this do the following
1) Check dropped packets from all interfaces. So if eth0/eth1 are connected to bond0, we may see dropped packets for bond0 and eth0, but not for eth1. This depends on which interface is the active interface. This can be checked using the following:
So if the
active_slave
isn't dropping packets, and the inactive slave is dropping packets this is normal in 'active-backup
' mode (or any mode where there is an inactive slave).2) If we want both interfaces to not drop packets we can use '
all_slaves_active
' bonding module parameter.Check:
It should default to 0 which means drop frames on the inactive slave.
If we set this to 1, we will no longer drop frames:
Kernel Documentation