Problem with subsequent confirmed uplinks if ack is not received

[Makhaos]

Describe your question or issue
Hi! Thank you for a great work with this library:)
Issue:
When I send a confirmed uplink from the node and the downlink ack is not received, the subsequent uplinks sent are being dropped by the gateway. Using LMIC_setTxData2.
The first uplink:

  "@type": "type.googleapis.com/ttn.lorawan.v3.UplinkMessage",
  "raw_payload": "gHaOAgAABQAKZza/GgX5hZTInkFpjQ8vbcLck1fYI13q",
  "payload": {
    "m_hdr": {
      "m_type": "CONFIRMED_UP"
    },
    "mic": "2CNd6g==",
    "mac_payload": {
      "f_hdr": {
        "dev_addr": "00028E76",
        "f_ctrl": {},
        "f_cnt": 5
      },
      "f_port": 10,
      "frm_payload": "Zza/GgX5hZTInkFpjQ8vbcLck1c="
    }
  },
  "settings": {
    "data_rate": {
      "lora": {
        "bandwidth": 125000,
        "spreading_factor": 7
      }
    },
    "coding_rate": "4/5",
    "frequency": "867500000",
    "timestamp": 3189063972,
    "time": "2020-08-03T15:28:38.639824Z"
  },
  "rx_metadata": [
    {
      "gateway_ids": {
        "gateway_id": "ic880a",
        "eui": "B827EBFFFE09CD2D"
      },
      "time": "2020-08-03T15:28:38.639824Z",
      "timestamp": 3189063972,
      "rssi": -53,
      "channel_rssi": -53,
      "snr": 7.8,
      "uplink_token": "ChQKEgoGaWM4ODBhEgi4J+v//gnNLRCkgtXwCw==",
      "channel_index": 5
    }
  ],
  "received_at": "2020-08-03T15:28:38.660948304Z",
  "correlation_ids": [
    "gs:conn:01EET7WDEXR73KVTZKMX0PTNZJ",
    "gs:uplink:01EETF0QR52GS6QV8ZJWYS7HEB"
  ]
}

The downlink sent by the gateway, which was not received by the node:

{
  "@type": "type.googleapis.com/ttn.lorawan.v3.DownlinkMessage",
  "raw_payload": "YHaOAgCgBQDJVCfQ",
  "scheduled": {
    "data_rate": {
      "lora": {
        "bandwidth": 125000,
        "spreading_factor": 7
      }
    },
    "data_rate_index": 5,
    "coding_rate": "4/5",
    "frequency": "867500000",
    "timestamp": 3194063972,
    "downlink": {
      "tx_power": 16.15,
      "invert_polarization": true
    }
  },
  "correlation_ids": [
    "gs:conn:01EET7WDEXR73KVTZKMX0PTNZJ",
    "gs:uplink:01EETF0QR52GS6QV8ZJWYS7HEB",
    "ns:downlink:01EETF0R67S03EHT5GQFKRY6M9",
    "ns:uplink:01EETF0QR6N67R189PYYESG7WE",
    "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01EETF0QR66KB9NYEX16CRRVCT",
    "gs:conn:01EET7WDEXR73KVTZKMX0PTNZJ",
    "rpc:/ttn.lorawan.v3.NsGs/ScheduleDownlink:01EETF0R67A38TGSZ5EZDVR0EE"
  ]
}

So then the node sends a second uplink:

{
  "@type": "type.googleapis.com/ttn.lorawan.v3.UplinkMessage",
  "raw_payload": "gHaOAgAABQAKZza/GgX5hZTInkFpjQ8vbcLck1fYI13q",
  "payload": {
    "m_hdr": {
      "m_type": "CONFIRMED_UP"
    },
    "mic": "2CNd6g==",
    "mac_payload": {
      "f_hdr": {
        "dev_addr": "00028E76",
        "f_ctrl": {},
        "f_cnt": 5
      },
      "f_port": 10,
      "frm_payload": "Zza/GgX5hZTInkFpjQ8vbcLck1c="
    }
  },
  "settings": {
    "data_rate": {
      "lora": {
        "bandwidth": 125000,
        "spreading_factor": 7
      }
    },
    "coding_rate": "4/5",
    "frequency": "867700000",
    "timestamp": 3196385604,
    "time": "2020-08-03T15:28:45.968042Z"
  },
  "rx_metadata": [
    {
      "gateway_ids": {
        "gateway_id": "ic880a",
        "eui": "B827EBFFFE09CD2D"
      },
      "time": "2020-08-03T15:28:45.968042Z",
      "timestamp": 3196385604,
      "rssi": -55,
      "channel_rssi": -55,
      "snr": 10,
      "uplink_token": "ChQKEgoGaWM4ODBhEgi4J+v//gnNLRDE8pP0Cw==",
      "channel_index": 6
    }
  ],
  "received_at": "2020-08-03T15:28:45.989152709Z",
  "correlation_ids": [
    "gs:conn:01EET7WDEXR73KVTZKMX0PTNZJ",
    "gs:uplink:01EETF0YX5TS7ZG5SKD4JAJ3Z7"
  ]
}

So then the gateway drops the uplink message with a Device not found error:

{
  "@type": "type.googleapis.com/ttn.lorawan.v3.ErrorDetails",
  "namespace": "pkg/gatewayserver",
  "name": "host_handle",
  "message_format": "host `{host}` failed to handle message",
  "attributes": {
    "host": "cluster"
  },
  "cause": {
    "namespace": "pkg/networkserver",
    "name": "device_not_found",
    "message_format": "device not found",
    "correlation_id": "1a5dd9f46d674b7a8edcce395360488a",
    "code": 5
  },
  "code": 5
}

This issue only happens sometimes, usually the node is able to receive the Ack.
The node sends 8 uplinks in total, until it triggers the EV_TXCOMPLETE, without receiving any ack.

Are there some details that should be updated like the frame counter?
I have tried to update the uplink sequence number, after EV_TXSTART with:

u4_t seq = LMIC_getSeqnoUp();
LMIC_setSeqnoUp(seq);

but with no success of the gateway to accept the subsequent messages.

I am joining the network with OTAA.

Is there something I am missing in my code, or should the library take this into account?

I have made a post on the TTN forum before, that you can read here:
TTN forum post

Environment

• Version of LMIC being used: 3.2.0
• Version of IDE being used: PlatformIO Core 4.3.4 Home 3.2.3
• Network provider: The Things Stack v3.8.6
• Region EU868
• Board TTGO-T-Beam v1.0
• Radio: SX1276
• Gateway: private iC880a-based gateway with a RPi.

[cstratton]

While this does seem to be disadvantageous behavior, it's not a "bug" but rather things happening in accordance with how the LoRaWAN spec says they should, eg, if there's a "bug", it's in the design of LoRaWAN, not the code.

One mitigation is to find the aspect of the code that limits how many times the node will try to uplink before giving up on that message - but this is wasteful, as if it misses the solitary ack that will be generated, it can never get another until it increments the uplink frame count, so any number of retries are wasted when it's the ack and not the uplink that got lost.

Typically the better approach is to ignore the fact that LoRaWAN even offers "confirmed uplink" and do your own confirmation at application level. Keep in mind that transmitting even just an ACK is very "expensive" of gateway capacity, so should only be used where you really need it. But if you are going to try to get an ACK, and then to resend, do you really want to be resending what is probably increasingly stale data? Or do you want to send new, fresher measurements? Doing your own ACKs at application level gives you the most flexibility to decide. It also implicitly means that you're acking the success of the complete path through to your data backend, not just between the node and the network server. And because each application level packet has a unique frame count, the network server won't start ignoring your retries but will rather pass each through to your data consumer.

[Makhaos]

Thank you for your detailed answer, I read something similar in another post, I just didn't understand it correctly back then.

I believe what you call stale data might be quite valuable, depending on the application, since the history of data that has been passed can be as important as fresh data. In any case, there is always the possibility to save the data and resend in a new packet.
Regarding the ACKs in the application level, it does sound good, to have a proper ACK from the complete path in the backend, however I am uncertain if it would be reliable regarding timings, in class A devices. The ACK might be sent too late for the device to be listening.
EDIT: Perhaps you meant that an ACK could be sent as a downlink for the next uplink of the device.

For now I think I will send a complete new packet with new data if the first was not ACK. It would be great to have an EVENT in the library that fires up if the RX windows are closed, after sending a confirmed uplink.

[cstratton]

Keep in mind you'll quickly burn up your TTN downlink allowance if you try to ack every uplink. If I recall, you are not supposed to send more than 10 downlinks a day, and yes, ack's count as downlinks, no matter if they are MAC level acks generated from the protocol, or application-level. The real cost to the network is taking a gateway out of receive mode to send a downlink at all, and sending the 13 bytes of protocol overhead; if you then send a byte or two of application data, or not, is scarcely relevant.

[terrillmoore]

It would be great to have an EVENT in the library that fires up if the RX windows are closed, after sending a confirmed uplink.

There are such events, but they're not sent on the low-level event API.. You should wait until you get the callback from a send-with-callback function. Until that happens, the LMIC is busy doing a transmit. You can't take TXCOMPLETE as meaning LMIC is idle; you have to look at all the flags (or better, use the callback version of the API which will check the calls for you).

Your description sounds like you might be trying to transmit while the LMIC is busy; it will just fail your request. You need to check the return code from the send API you're happening to use.