You are here

Dropped Packets on MariaDB/MySQL Linux Servers

Table of Contents


When we do MariaDB and MySQL Database server analysis we see from time to time systems with a lot of dropped packets on a network interface:

ifconfig

# export DEV=enp0s25
# ifconfig ${DEV}
eth0 Link encap:Ethernet  HWaddr d0:50:99:70:f8:8e
     inet addr:192.168.1.135  Bcast:192.168.1.255  Mask:255.255.255.0
     UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     RX packets:4005096 errors:0 dropped:365956 overruns:0 frame:0
     TX packets:3293230 errors:0 dropped:0 overruns:0 carrier:0
     collisions:0 txqueuelen:1000
     RX bytes:2897493579 (2.8 GB)  TX bytes:461542567 (461.5 MB)

As we all learned ifconfig is old and thus bad so we should use the new command ip instead. See also our new hint: Replacement for ifconfig statistics.

ip

# ip --statistics [--human-readable] address show ${DEV}

2: eth0:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether d0:50:99:70:f8:8e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.135/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    2897493579 4005096  0       365956  0       19703   
    TX: bytes  packets  errors  dropped carrier collsns 
    461542567  3293230  0       0       0       0       

ethtool --statistics

If you want to see some more statistics about your packets you can run:

# ethtool --statistics ${DEV}
NIC statistics:
     rx_packets: 4134
     tx_packets: 3507
     rx_bytes: 2791271
     tx_bytes: 607408
     rx_broadcast: 59
     tx_broadcast: 82
     rx_multicast: 89
     tx_multicast: 85
     rx_errors: 0
     tx_errors: 0
     tx_dropped: 0
     multicast: 89
     collisions: 0
     rx_length_errors: 0
     rx_over_errors: 0
     rx_crc_errors: 0
     rx_frame_errors: 0
     rx_no_buffer_count: 0
     rx_missed_errors: 0
     tx_aborted_errors: 0
     tx_carrier_errors: 0
     tx_fifo_errors: 0
     tx_heartbeat_errors: 0
     tx_window_errors: 0
     tx_abort_late_coll: 0
     tx_deferred_ok: 0
     tx_single_coll_ok: 0
     tx_multi_coll_ok: 0
     tx_timeout_count: 0
     tx_restart_queue: 0
     rx_long_length_errors: 0
     rx_short_length_errors: 0
     rx_align_errors: 0
     tx_tcp_seg_good: 53
     tx_tcp_seg_failed: 0
     rx_flow_control_xon: 0
     rx_flow_control_xoff: 0
     tx_flow_control_xon: 0
     tx_flow_control_xoff: 0
     rx_csum_offload_good: 3983
     rx_csum_offload_errors: 0
     rx_header_split: 0
     alloc_rx_buff_failed: 0
     tx_smbus: 0
     rx_smbus: 53
     dropped_smbus: 0
     rx_dma_failed: 0
     tx_dma_failed: 0
     rx_hwtstamp_cleared: 0
     uncorr_ecc_errors: 0
     corr_ecc_errors: 0
     tx_hwtstamp_timeouts: 0
     tx_hwtstamp_skipped: 0

But unfortunately RX dropped is not listed but only TX dropped.

nstat

Detailed network statistics should be checked with nstat today:

# nstat --ignore --noupdate --zero
#kernel
SctpCurrEstab                   0                  0.0
SctpActiveEstabs                0                  0.0
SctpPassiveEstabs               0                  0.0
SctpAborteds                    0                  0.0
SctpShutdowns                   0                  0.0
SctpOutOfBlues                  0                  0.0
SctpChecksumErrors              0                  0.0
SctpOutCtrlChunks               0                  0.0
SctpOutOrderChunks              0                  0.0
SctpOutUnorderChunks            0                  0.0
SctpInCtrlChunks                0                  0.0
SctpInOrderChunks               0                  0.0
SctpInUnorderChunks             0                  0.0
SctpFragUsrMsgs                 0                  0.0
SctpReasmUsrMsgs                0                  0.0
SctpOutSCTPPacks                0                  0.0
SctpInSCTPPacks                 0                  0.0
SctpT1InitExpireds              0                  0.0
SctpT1CookieExpireds            0                  0.0
SctpT2ShutdownExpireds          0                  0.0
SctpT3RtxExpireds               0                  0.0
SctpT4RtoExpireds               0                  0.0
SctpT5ShutdownGuardExpireds     0                  0.0
SctpDelaySackExpireds           0                  0.0
SctpAutocloseExpireds           0                  0.0
SctpT3Retransmits               0                  0.0
SctpPmtudRetransmits            0                  0.0
SctpFastRetransmits             0                  0.0
SctpInPktSoftirq                0                  0.0
SctpInPktBacklog                0                  0.0
SctpInPktDiscards               0                  0.0
SctpInDataChunkDiscards         0                  0.0
IpInReceives                    2146596            0.0
IpInHdrErrors                   0                  0.0
IpInAddrErrors                  2                  0.0
IpForwDatagrams                 0                  0.0
IpInUnknownProtos               0                  0.0
IpInDiscards                    0                  0.0
IpInDelivers                    2146589            0.0
IpOutRequests                   1821062            0.0
IpOutDiscards                   20                 0.0
IpOutNoRoutes                   3                  0.0
IpReasmTimeout                  0                  0.0
IpReasmReqds                    6                  0.0
IpReasmOKs                      3                  0.0
IpReasmFails                    0                  0.0
IpFragOKs                       0                  0.0
IpFragFails                     0                  0.0
IpFragCreates                   0                  0.0
IcmpInMsgs                      7434               0.0
IcmpInErrors                    14                 0.0
IcmpInCsumErrors                0                  0.0
IcmpInDestUnreachs              3703               0.0
IcmpInTimeExcds                 0                  0.0
IcmpInParmProbs                 0                  0.0
IcmpInSrcQuenchs                0                  0.0
IcmpInRedirects                 0                  0.0
IcmpInEchos                     610                0.0
IcmpInEchoReps                  3121               0.0
IcmpInTimestamps                0                  0.0
IcmpInTimestampReps             0                  0.0
IcmpInAddrMasks                 0                  0.0
IcmpInAddrMaskReps              0                  0.0
IcmpOutMsgs                     12373              0.0
IcmpOutErrors                   0                  0.0
IcmpOutDestUnreachs             3701               0.0
IcmpOutTimeExcds                0                  0.0
IcmpOutParmProbs                0                  0.0
IcmpOutSrcQuenchs               0                  0.0
IcmpOutRedirects                0                  0.0
IcmpOutEchos                    8062               0.0
IcmpOutEchoReps                 610                0.0
IcmpOutTimestamps               0                  0.0
IcmpOutTimestampReps            0                  0.0
IcmpOutAddrMasks                0                  0.0
IcmpOutAddrMaskReps             0                  0.0
IcmpMsgInType0                  3121               0.0
IcmpMsgInType3                  3703               0.0
IcmpMsgInType8                  610                0.0
IcmpMsgOutType0                 610                0.0
IcmpMsgOutType3                 3701               0.0
IcmpMsgOutType8                 8062               0.0
TcpActiveOpens                  31370              0.0
TcpPassiveOpens                 18080              0.0
TcpAttemptFails                 2997               0.0
TcpEstabResets                  56                 0.0
TcpInSegs                       2166708            0.0
TcpOutSegs                      1842768            0.0
TcpRetransSegs                  4093               0.0
TcpInErrs                       5                  0.0
TcpOutRsts                      2905               0.0
TcpInCsumErrors                 0                  0.0
UdpInDatagrams                  40549              0.0
UdpNoPorts                      41                 0.0
UdpInErrors                     0                  0.0
UdpOutDatagrams                 38161              0.0
UdpRcvbufErrors                 0                  0.0
UdpSndbufErrors                 0                  0.0
UdpInCsumErrors                 0                  0.0
UdpIgnoredMulti                 181                0.0
UdpLiteInDatagrams              0                  0.0
UdpLiteNoPorts                  0                  0.0
UdpLiteInErrors                 0                  0.0
UdpLiteOutDatagrams             0                  0.0
UdpLiteRcvbufErrors             0                  0.0
UdpLiteSndbufErrors             0                  0.0
UdpLiteInCsumErrors             0                  0.0
UdpLiteIgnoredMulti             0                  0.0
Ip6InReceives                   71921              0.0
Ip6InHdrErrors                  0                  0.0
Ip6InTooBigErrors               0                  0.0
Ip6InNoRoutes                   1                  0.0
Ip6InAddrErrors                 0                  0.0
Ip6InUnknownProtos              0                  0.0
Ip6InTruncatedPkts              0                  0.0
Ip6InDiscards                   0                  0.0
Ip6InDelivers                   71920              0.0
Ip6OutForwDatagrams             0                  0.0
Ip6OutRequests                  68903              0.0
Ip6OutDiscards                  0                  0.0
Ip6OutNoRoutes                  47                 0.0
Ip6ReasmTimeout                 0                  0.0
Ip6ReasmReqds                   0                  0.0
Ip6ReasmOKs                     0                  0.0
Ip6ReasmFails                   0                  0.0
Ip6FragOKs                      0                  0.0
Ip6FragFails                    0                  0.0
Ip6FragCreates                  0                  0.0
Ip6InMcastPkts                  2060               0.0
Ip6OutMcastPkts                 94                 0.0
Ip6InOctets                     80749746           0.0
Ip6OutOctets                    9844802            0.0
Ip6InMcastOctets                247993             0.0
Ip6OutMcastOctets               12725              0.0
Ip6InBcastOctets                0                  0.0
Ip6OutBcastOctets               0                  0.0
Ip6InNoECTPkts                  103468             0.0
Ip6InECT1Pkts                   1                  0.0
Ip6InECT0Pkts                   0                  0.0
Ip6InCEPkts                     1                  0.0
Icmp6InMsgs                     3054               0.0
Icmp6InErrors                   0                  0.0
Icmp6OutMsgs                    1948               0.0
Icmp6OutErrors                  0                  0.0
Icmp6InCsumErrors               0                  0.0
Icmp6InDestUnreachs             0                  0.0
Icmp6InPktTooBigs               0                  0.0
Icmp6InTimeExcds                0                  0.0
Icmp6InParmProblems             0                  0.0
Icmp6InEchos                    776                0.0
Icmp6InEchoReplies              0                  0.0
Icmp6InGroupMembQueries         294                0.0
Icmp6InGroupMembResponses       0                  0.0
Icmp6InGroupMembReductions      0                  0.0
Icmp6InRouterSolicits           0                  0.0
Icmp6InRouterAdvertisements     836                0.0
Icmp6InNeighborSolicits         684                0.0
Icmp6InNeighborAdvertisements   462                0.0
Icmp6InRedirects                0                  0.0
Icmp6InMLDv2Reports             2                  0.0
Icmp6OutDestUnreachs            0                  0.0
Icmp6OutPktTooBigs              0                  0.0
Icmp6OutTimeExcds               0                  0.0
Icmp6OutParmProblems            0                  0.0
Icmp6OutEchos                   0                  0.0
Icmp6OutEchoReplies             776                0.0
Icmp6OutGroupMembQueries        0                  0.0
Icmp6OutGroupMembResponses      0                  0.0
Icmp6OutGroupMembReductions     0                  0.0
Icmp6OutRouterSolicits          1                  0.0
Icmp6OutRouterAdvertisements    0                  0.0
Icmp6OutNeighborSolicits        464                0.0
Icmp6OutNeighborAdvertisements  684                0.0
Icmp6OutRedirects               0                  0.0
Icmp6OutMLDv2Reports            23                 0.0
Icmp6InType128                  776                0.0
Icmp6InType130                  294                0.0
Icmp6InType134                  836                0.0
Icmp6InType135                  684                0.0
Icmp6InType136                  462                0.0
Icmp6InType143                  2                  0.0
Icmp6OutType129                 776                0.0
Icmp6OutType133                 1                  0.0
Icmp6OutType135                 464                0.0
Icmp6OutType136                 684                0.0
Icmp6OutType143                 23                 0.0
Udp6InDatagrams                 248                0.0
Udp6NoPorts                     0                  0.0
Udp6InErrors                    0                  0.0
Udp6OutDatagrams                153                0.0
Udp6RcvbufErrors                0                  0.0
Udp6SndbufErrors                0                  0.0
Udp6InCsumErrors                0                  0.0
Udp6IgnoredMulti                0                  0.0
UdpLite6InDatagrams             0                  0.0
UdpLite6NoPorts                 0                  0.0
UdpLite6InErrors                0                  0.0
UdpLite6OutDatagrams            0                  0.0
UdpLite6RcvbufErrors            0                  0.0
UdpLite6SndbufErrors            0                  0.0
UdpLite6InCsumErrors            0                  0.0
TcpExtSyncookiesSent            0                  0.0
TcpExtSyncookiesRecv            0                  0.0
TcpExtSyncookiesFailed          0                  0.0
TcpExtEmbryonicRsts             0                  0.0
TcpExtPruneCalled               0                  0.0
TcpExtRcvPruned                 0                  0.0
TcpExtOfoPruned                 0                  0.0
TcpExtOutOfWindowIcmps          0                  0.0
TcpExtLockDroppedIcmps          0                  0.0
TcpExtArpFilter                 0                  0.0
TcpExtTW                        20345              0.0
TcpExtTWRecycled                0                  0.0
TcpExtTWKilled                  0                  0.0
TcpExtPAWSActive                0                  0.0
TcpExtPAWSEstab                 2                  0.0
TcpExtDelayedACKs               15602              0.0
TcpExtDelayedACKLocked          9                  0.0
TcpExtDelayedACKLost            114                0.0
TcpExtListenOverflows           0                  0.0
TcpExtListenDrops               0                  0.0
TcpExtTCPHPHits                 1591674            0.0
TcpExtTCPPureAcks               96024              0.0
TcpExtTCPHPAcks                 92591              0.0
TcpExtTCPRenoRecovery           0                  0.0
TcpExtTCPSackRecovery           3                  0.0
TcpExtTCPSACKReneging           0                  0.0
TcpExtTCPSACKReorder            84                 0.0
TcpExtTCPRenoReorder            0                  0.0
TcpExtTCPTSReorder              0                  0.0
TcpExtTCPFullUndo               0                  0.0
TcpExtTCPPartialUndo            0                  0.0
TcpExtTCPDSACKUndo              0                  0.0
TcpExtTCPLossUndo               7                  0.0
TcpExtTCPLostRetransmit         1                  0.0
TcpExtTCPRenoFailures           0                  0.0
TcpExtTCPSackFailures           0                  0.0
TcpExtTCPLossFailures           0                  0.0
TcpExtTCPFastRetrans            31                 0.0
TcpExtTCPSlowStartRetrans       0                  0.0
TcpExtTCPTimeouts               3774               0.0
TcpExtTCPLossProbes             69                 0.0
TcpExtTCPLossProbeRecovery      3                  0.0
TcpExtTCPRenoRecoveryFail       0                  0.0
TcpExtTCPSackRecoveryFail       0                  0.0
TcpExtTCPRcvCollapsed           0                  0.0
TcpExtTCPDSACKOldSent           109                0.0
TcpExtTCPDSACKOfoSent           1                  0.0
TcpExtTCPDSACKRecv              25                 0.0
TcpExtTCPDSACKOfoRecv           0                  0.0
TcpExtTCPAbortOnData            548                0.0
TcpExtTCPAbortOnClose           29                 0.0
TcpExtTCPAbortOnMemory          0                  0.0
TcpExtTCPAbortOnTimeout         27                 0.0
TcpExtTCPAbortOnLinger          0                  0.0
TcpExtTCPAbortFailed            0                  0.0
TcpExtTCPMemoryPressures        0                  0.0
TcpExtTCPMemoryPressuresChrono  0                  0.0
TcpExtTCPSACKDiscard            0                  0.0
TcpExtTCPDSACKIgnoredOld        0                  0.0
TcpExtTCPDSACKIgnoredNoUndo     18                 0.0
TcpExtTCPSpuriousRTOs           0                  0.0
TcpExtTCPMD5NotFound            0                  0.0
TcpExtTCPMD5Unexpected          0                  0.0
TcpExtTCPMD5Failure             0                  0.0
TcpExtTCPSackShifted            28                 0.0
TcpExtTCPSackMerged             20                 0.0
TcpExtTCPSackShiftFallback      95                 0.0
TcpExtTCPBacklogDrop            0                  0.0
TcpExtPFMemallocDrop            0                  0.0
TcpExtTCPMinTTLDrop             0                  0.0
TcpExtTCPDeferAcceptDrop        1                  0.0
TcpExtIPReversePathFilter       0                  0.0
TcpExtTCPTimeWaitOverflow       0                  0.0
TcpExtTCPReqQFullDoCookies      0                  0.0
TcpExtTCPReqQFullDrop           0                  0.0
TcpExtTCPRetransFail            0                  0.0
TcpExtTCPRcvCoalesce            200321             0.0
TcpExtTCPOFOQueue               61316              0.0
TcpExtTCPOFODrop                0                  0.0
TcpExtTCPOFOMerge               2                  0.0
TcpExtTCPChallengeACK           5                  0.0
TcpExtTCPSYNChallenge           5                  0.0
TcpExtTCPFastOpenActive         0                  0.0
TcpExtTCPFastOpenActiveFail     0                  0.0
TcpExtTCPFastOpenPassive        0                  0.0
TcpExtTCPFastOpenPassiveFail    0                  0.0
TcpExtTCPFastOpenListenOverflow 0                  0.0
TcpExtTCPFastOpenCookieReqd     0                  0.0
TcpExtTCPFastOpenBlackhole      0                  0.0
TcpExtTCPSpuriousRtxHostQueues  12                 0.0
TcpExtBusyPollRxPackets         0                  0.0
TcpExtTCPAutoCorking            4012               0.0
TcpExtTCPFromZeroWindowAdv      0                  0.0
TcpExtTCPToZeroWindowAdv        0                  0.0
TcpExtTCPWantZeroWindowAdv      188                0.0
TcpExtTCPSynRetrans             3737               0.0
TcpExtTCPOrigDataSent           437079             0.0
TcpExtTCPHystartTrainDetect     35                 0.0
TcpExtTCPHystartTrainCwnd       1153               0.0
TcpExtTCPHystartDelayDetect     0                  0.0
TcpExtTCPHystartDelayCwnd       0                  0.0
TcpExtTCPACKSkippedSynRecv      0                  0.0
TcpExtTCPACKSkippedPAWS         0                  0.0
TcpExtTCPACKSkippedSeq          10                 0.0
TcpExtTCPACKSkippedFinWait2     0                  0.0
TcpExtTCPACKSkippedTimeWait     0                  0.0
TcpExtTCPACKSkippedChallenge    0                  0.0
TcpExtTCPWinProbe               0                  0.0
TcpExtTCPKeepAlive              5134               0.0
TcpExtTCPMTUPFail               0                  0.0
TcpExtTCPMTUPSuccess            0                  0.0
TcpExtTCPWqueueTooBig           0                  0.0
IpExtInNoRoutes                 0                  0.0
IpExtInTruncatedPkts            0                  0.0
IpExtInMcastPkts                2893               0.0
IpExtOutMcastPkts               466                0.0
IpExtInBcastPkts                190                0.0
IpExtOutBcastPkts               15                 0.0
IpExtInOctets                   4837352803         0.0
IpExtOutOctets                  1199138713         0.0
IpExtInMcastOctets              2244078            0.0
IpExtOutMcastOctets             40977              0.0
IpExtInBcastOctets              45209              0.0
IpExtOutBcastOctets             4035               0.0
IpExtInCsumErrors               0                  0.0
IpExtInNoECTPkts                3140592            0.0
IpExtInECT1Pkts                 0                  0.0
IpExtInECT0Pkts                 0                  0.0
IpExtInCEPkts                   0                  0.0
IpExtReasmOverlaps              0                  0.0

netstat --statistics

Old command netstat but still widely used an known (use the previous nstat instead):

# netstat --statistics ${DEV}
TcpExt:
    65 invalid SYN cookies received
    155 resets received for embryonic SYN_RECV sockets
    89 packets pruned from receive queue because of socket buffer overrun
    ArpFilter: 0
    14341 TCP sockets finished time wait in fast timer
    10 time wait sockets recycled by time stamp
    713178 delayed acks sent
    316 delayed acks further delayed because of locked socket
    Quick ack mode was activated 3908 times
    29091413 packets directly queued to recvmsg prequeue.
    19881778 packets directly received from backlog
    1892094685 packets directly received from prequeue
    31118010 packets header predicted
    1272468 packets header predicted and directly queued to user
    TCPPureAcks: 1118011
    TCPHPAcks: 30188520
    TCPRenoRecovery: 2524
    TCPSackRecovery: 1
    TCPSACKReneging: 0
    TCPFACKReorder: 0
    TCPSACKReorder: 0
    TCPRenoReorder: 277
    TCPTSReorder: 128
    TCPFullUndo: 1038
    TCPPartialUndo: 640
    TCPDSACKUndo: 0
    TCPLossUndo: 1
    TCPLoss: 0
    TCPLostRetransmit: 0
    TCPRenoFailures: 117
    TCPSackFailures: 0
    TCPLossFailures: 42
    TCPFastRetrans: 1991
    TCPForwardRetrans: 0
    TCPSlowStartRetrans: 668
    TCPTimeouts: 3276
    TCPRenoRecoveryFail: 674
    TCPSackRecoveryFail: 0
    TCPSchedulerFailed: 2
    TCPRcvCollapsed: 5439
    TCPDSACKOldSent: 4
    TCPDSACKOfoSent: 0
    TCPDSACKRecv: 2
    TCPDSACKOfoRecv: 0
    TCPAbortOnSyn: 0
    TCPAbortOnData: 14
    TCPAbortOnClose: 56
    TCPAbortOnMemory: 0
    TCPAbortOnTimeout: 32
    TCPAbortOnLinger: 0
    TCPAbortFailed: 0
    TCPMemoryPressures: 0

ss

Current replacement of netstat is ss:

# ss --summary
Total: 1449 (kernel 3546)
TCP:   79 (estab 19, closed 33, orphaned 0, synrecv 0, timewait 31/0), ports 0

Transport Total     IP        IPv6
*         3546      -         -        
RAW       0         0         0        
UDP       9         6         3        
TCP       46        16        30       
INET      55        22        33       
FRAG      0         0         0        

Reasons for dropped packets

  • NIC ring buffers getting full and unable to cope-up with incoming bursts of traffic
  • CPU receiving NIC interrupts is very busy and unable to process
  • Some cable/hardware/duplex issues
  • Some bug in NIC driver
  • Softnet backlog full (Measured from /proc/net/softnet_stat)
  • Bad / Unintended VLAN tags
  • Unknown / Unregistered protocols
  • IPv6 frames when the server is not configured for IPv6
  • Possibly some miss-configuration in Hypervisor (VMware)?
  • Missing flow control?

Features of Network card

# ethtool ${DEV}
Settings for enp0s25:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Full 
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Full 
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: on (auto)
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

Features of Network driver

# ethtool --show-features ${DEV}
Features for enp0s25:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]

Network driver ring buffer

# ethtool --show-ring ${DEV}
Ring parameters for enp0s25:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             256
RX Mini:        0
RX Jumbo:       0
TX:             256

# ethtool --set-ring ${DEV} rx 4096 

# ethtool --show-ring ${DEV}
Ring parameters for enp0s25:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             256

This did not help in my case.

TCP socket received and send buffer

# /sbin/sysctl -a --pattern 'net.ipv4' | grep mem
net.ipv4.igmp_max_memberships = 20
net.ipv4.tcp_mem = 186753       249007  373506
net.ipv4.tcp_rmem = 4096        87380   6291456
net.ipv4.tcp_wmem = 4096        16384   4194304
net.ipv4.udp_mem = 373509       498014  747018
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096

Flow Control

If flow control is activated can be checked as follows:

# ethtool -a ${DEV}
Pause parameters for enp0s25:
Autonegotiate:  on
RX:             on
TX:             on

Check over time: sar

To check the behaviour over time sar is a good tool:

# sar -n EDEV 1 | grep -v lo
Linux 4.15.0-70-generic (chef)  28/11/19        _x86_64_        (4 CPU)

11:56:51        IFACE   rxerr/s   txerr/s    coll/s  rxdrop/s  txdrop/s  txcarr/s  rxfram/s  rxfifo/s  txfifo/s
11:56:52      enp0s25      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:56:53      enp0s25      0.00      0.00      0.00      2.00      0.00      0.00      0.00      0.00      0.00
11:56:54      enp0s25      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:56:55      enp0s25      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Dropwatch

# git clone https://github.com/pavel-odintsov/drop_watch
# cd drop_watch/src
# make
gcc -I/usr/include/libnl3/  -c -g -D_GNU_SOURCE -Wall -Werror main.c
gcc -I/usr/include/libnl3/  -c -g -D_GNU_SOURCE -Wall -Werror lookup.c
gcc -I/usr/include/libnl3/  -c -g -D_GNU_SOURCE -Wall -Werror lookup_bfd.c
gcc -I/usr/include/libnl3/  -c -g -D_GNU_SOURCE -Wall -Werror lookup_kas.c
gcc -I/usr/include/libnl3/ -g -o dropwatch main.o lookup.o lookup_bfd.o lookup_kas.o -lbfd -lreadline -lnl-3 -lnl-genl-3

# ./dropwatch -l kas
Initalizing kallsyms db
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
4 drops at sk_stream_kill_queues+4d (0xffffffff8c05c22d)
1 drops at inet_sock_destruct+4d (0xffffffff8c0f9e8d)
2 drops at arp_error_report+41 (0xffffffff8c0f00f1)
4 drops at unix_stream_connect+2e5 (0xffffffff8c12bbf5)
1 drops at neigh_probe+61 (0xffffffff8c076e61)
1 drops at sk_stream_kill_queues+4d (0xffffffff8c05c22d)
9 drops at skb_queue_purge+17 (0xffffffff8c0550f7)
2 drops at neigh_probe+61 (0xffffffff8c076e61)
4 drops at unix_stream_connect+2e5 (0xffffffff8c12bbf5)
3 drops at sk_stream_kill_queues+4d (0xffffffff8c05c22d)
2 drops at arp_error_report+41 (0xffffffff8c0f00f1)
1 drops at icmpv6_rcv+1c1 (0xffffffff8c159081)
9 drops at skb_queue_purge+17 (0xffffffff8c0550f7)
2 drops at __netif_receive_skb_core+6b8 (0xffffffff8c06a498)
2 drops at __netif_receive_skb_core+6b8 (0xffffffff8c06a498)
9 drops at skb_queue_purge+17 (0xffffffff8c0550f7)
9 drops at skb_queue_purge+17 (0xffffffff8c0550f7)
9 drops at skb_queue_purge+17 (0xffffffff8c0550f7)
9 drops at skb_queue_purge+17 (0xffffffff8c0550f7)
9 drops at skb_queue_purge+17 (0xffffffff8c0550f7)
4 drops at icmpv6_rcv+1c1 (0xffffffff8c159081)
9 drops at skb_queue_purge+17 (0xffffffff8c0550f7)
Got a stop message
dropwatch> Terminating dropwatch...

Some probing led to the function: __netif_receive_skb_core+6b8. But no clue what it is.

perf

perf record -g -a -e skb:kfree_skb
perf script
...
swapper     0 [003]  3161.113873: skb:kfree_skb: skbaddr=0xffff9247df562000 protocol=34939 location=0xffffffffb586a498
            7fffb5856ec6 kfree_skb ([kernel.kallsyms])
            7fffb586c498 __netif_receive_skb_core ([kernel.kallsyms])
            7fffb586c998 __netif_receive_skb ([kernel.kallsyms])
            7fffb586ed25 netif_receive_skb_internal ([kernel.kallsyms])
            7fffb586faa5 napi_gro_receive ([kernel.kallsyms])
            7fffc01fa9c6 e1000_receive_skb ([kernel.kallsyms])
            7fffc01fc58e e1000_clean_rx_irq ([kernel.kallsyms])
            7fffc0203fee e1000e_poll ([kernel.kallsyms])
            7fffb586f180 net_rx_action ([kernel.kallsyms])
            7fffb5c020e4 __softirqentry_text_start ([kernel.kallsyms])
            7fffb5098bd5 irq_exit ([kernel.kallsyms])
            7fffb5a04736 do_IRQ ([kernel.kallsyms])
            7fffb5a0290c ret_from_intr ([kernel.kallsyms])
            7fffb5806337 cpuidle_enter_state ([kernel.kallsyms])
            7fffb58065b7 cpuidle_enter ([kernel.kallsyms])
            7fffb50d96c3 call_cpuidle ([kernel.kallsyms])
            7fffb50d993c do_idle ([kernel.kallsyms])
            7fffb50d9b83 cpu_startup_entry ([kernel.kallsyms])
            7fffb505c61b start_secondary ([kernel.kallsyms])
            7fffb50020d5 secondary_startup_64 ([kernel.kallsyms])
...

tcpdump

# tcpdump --interface=${DEV} -w trace.pcap
tcpdump: listening on enp0s25, link-type EN10MB (Ethernet), capture size 262144 bytes
642 packets captured
642 packets received by filter
0 packets dropped by kernel
5 packets dropped by interface

# tcpdump -n -t -i ${DEV}
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s25, link-type EN10MB (Ethernet), capture size 262144 bytes
STP 802.1d, Config, Flags [none], bridge-id 8000.cc:5d:4e:2e:37:94.8002, length 47
IP chef > static.211.20.9.176.clients.your-server.de: ICMP echo request, id 7432, seq 1, length 64
IP chef.45753 > internetbox.home.domain: 51306+ [1au] PTR? 211.20.9.176.in-addr.arpa. (54)
IP static.211.20.9.176.clients.your-server.de > chef: ICMP echo reply, id 7432, seq 1, length 64
IP chef.51876 > static.211.20.9.176.clients.your-server.de.2222: Flags [S], seq 2727621364, win 29200, options [mss 1460,sackOK,TS val 1819633171 ecr 0,nop,wscale 7], length 0
IP internetbox.home.domain > chef.45753: 51306 1/0/1 PTR static.211.20.9.176.clients.your-server.de. (110)
IP chef.58204 > internetbox.home.domain: 37314+ [1au] PTR? 1.1.168.192.in-addr.arpa. (53)
IP internetbox.home.domain > chef.58204: 37314* 1/0/1 PTR internetbox.home. (83)
IP static.211.20.9.176.clients.your-server.de.2222 > chef.51876: Flags [R.], seq 0, ack 2727621365, win 0, length 0
IP chef > static.173.10.47.78.clients.your-server.de: ICMP echo request, id 7436, seq 1, length 64
IP chef.40629 > internetbox.home.domain: 32783+ [1au] PTR? 173.10.47.78.in-addr.arpa. (54)
IP static.173.10.47.78.clients.your-server.de > chef: ICMP echo reply, id 7436, seq 1, length 64
IP chef.34262 > static.173.10.47.78.clients.your-server.de.2222: Flags [S], seq 2923371146, win 29200, options [mss 1460,sackOK,TS val 1741234319 ecr 0,nop,wscale 7], length 0
IP internetbox.home.domain > chef.40629: 32783 1/0/1 PTR static.173.10.47.78.clients.your-server.de. (110)
e8:d1:1b:ec:98:30 (oui Unknown) > Broadcast, ethertype Unknown (0x88e1), length 60: 
        0x0000:  0120 6000 0000 0000 0000 0000 0000 0000  ..`.............
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............
e8:d1:1b:ec:98:31 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: 
        0x0000:  0120 6000 0000 0000 0000 0000 0000 0000  ..`.............
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............
IP chef.34262 > static.173.10.47.78.clients.your-server.de.2222: Flags [S], seq 2923371146, win 29200, options [mss 1460,sackOK,TS val 1741235343 ecr 0,nop,wscale 7], length 0
IP internetbox.home.mdns > 224.0.0.251.mdns: 0 [25a] [13q] PTR (QM)? _services._dns-sd._udp.local. PTR (QM)? _printer._tcp.local. PTR (QM)? _pdl-datastream._tcp.local. PTR (QM)? _ipp._tcp.local. PTR (QM)? _http._tcp.local. PTR (QM)? _scanner._tcp.local. PTR (QM)? _http-alt._tcp.local. PTR (QM)? _uscan._tcp.local. PTR (QM)? _ipps._tcp.local. PTR (QM)? _privet._tcp.local. PTR (QM)? _uscans._tcp.local. PTR (QM)? _mqtt._tcp.local. PTR (QM)? _ssw._tcp.local. (968)
IP chef.43103 > internetbox.home.domain: 55791+ [1au] PTR? 251.0.0.224.in-addr.arpa. (53)
IP internetbox.home.domain > chef.43103: 55791 NXDomain 0/1/1 (110)
IP chef.43103 > internetbox.home.domain: 55791+ PTR? 251.0.0.224.in-addr.arpa. (42)
IP internetbox.home.domain > chef.43103: 55791 NXDomain 0/0/0 (42)
STP 802.1d, Config, Flags [none], bridge-id 8000.cc:5d:4e:2e:37:94.8002, length 47
23 packets captured
23 packets received by filter
0 packets dropped by kernel
1 packet dropped by interface

/proc/net/netstat

# ./myenv/utl/netstat.php --skip-zeroj | grep -i drop
  LockDroppedIcmps                      0
  ListenDrops                           0
  TCPBacklogDrop                        0
  PFMemallocDrop                        0
  TCPMinTTLDrop                         0
  TCPDeferAcceptDrop                    0
  TCPReqQFullDrop                       0
  TCPOFODrop                            0

System Tap (stap)

# stap --all-modules dropwatch.stp 
Monitoring for dropped packets
18 packets dropped at skb_queue_purge
8 packets dropped at unix_stream_connect
5 packets dropped at sk_stream_kill_queues
2 packets dropped at icmp_rcv
2 packets dropped at __netif_receive_skb_core
1 packets dropped at tcp_v4_do_rcv
1 packets dropped at tcp_v4_rcv
Stopping dropped packet monitor

# stap --all-modules drop.stp 
Timestamp                Src IP  Dst IP        IP Proto TCP Src Port TCP Dst Port Execname
Wed Nov 27 21:55:51 2019 0.0.0.0 0.128.239.209     0     0     0 swapper/3
Wed Nov 27 21:55:51 2019 0.0.0.0 0.128.239.209     0     0     0 swapper/3
Wed Nov 27 21:56:06 2019 0.0.0.0 0.128.239.209     0     0     0 swapper/3
Wed Nov 27 21:56:06 2019 0.0.0.0 0.128.239.209     0     0     0 swapper/3
Wed Nov 27 21:56:36 2019 0.0.0.0 0.128.239.209     0     0     0 swapper/3
Wed Nov 27 21:56:36 2019 0.0.0.0 0.128.239.209     0     0     0 swapper/3

#! /usr/bin/env stap

#
# drop.stp
#

probe begin {
  printf("Timestamp                Src IP  Dst IP        IP Proto TCP Src Port TCP Dst Port Execname\n")
}
probe end {
  printf("Stopping dropped packet monitor\n")
}

probe kernel.trace("kfree_skb") {

  if (symname($location) == "__netif_receive_skb_core") {

    printf("%s %s %s %5d %5d %5d %s\n", ctime(gettimeofday_s()),
    ip_ntop(__ip_skb_saddr($skb)),
    ip_ntop(__ip_skb_daddr($skb)),
    __ip_skb_proto($skb),
    __tcp_skb_dport($skb),
    __tcp_skb_sport($skb),
    execname() )
  }
}

If interpretation of IP protocol is correctly:

0x000HOPOPTIPv6Hop-by-Hop OptionRFC 8200

No clue, what it finally is. Giving up...

Literature


Comments

We have found a strong correlation between VMware snapshots and Veeam backups and those dropped packets.
Shinguzcomment

802.3ad bond interface have show high RX dropped packets

I found a god explanation for dropped packages on bonded interfaces: [ 1 ]

This is related to the bonding mode and _not_ a bug. The bonding module will drop duplicate frames received on inactive ports, which is normal behaviour. Overall the packets should be getting into the machine without problems since they are received on the active slave. To confirm this do the following

1) Check dropped packets from all interfaces. So if eth0/eth1 are connected to bond0, we may see dropped packets for bond0 and eth0, but not for eth1. This depends on which interface is the active interface. This can be checked using the following:

cat /sys/class/net/bond0/bonding/active_slave

So if the active_slave isn't dropping packets, and the inactive slave is dropping packets this is normal in 'active-backup' mode (or any mode where there is an inactive slave).

2) If we want both interfaces to not drop packets we can use 'all_slaves_active' bonding module parameter.
Check:

cat /sys/class/net/bond0/bonding/all_slaves_active

It should default to 0 which means drop frames on the inactive slave.

If we set this to 1, we will no longer drop frames:

echo 1 | sudo tee /sys/class/net/bond0/bonding/all_slaves_active

Shinguzcomment

And this is what the Linux kernel documentation says about it: https://www.kernel.org/doc/Documentation/networking/bonding.txt
Shinguzcomment