From 9e4ee9c6ad275a672d1edd2df991ca666d219ce5 Mon Sep 17 00:00:00 2001 From: iceman1001 Date: Tue, 15 Sep 2026 22:49:41 +0200 Subject: [PATCH] lf read: fix realtime sample streaming over USB A realtime `lf read` streams samples straight to the host at the LF sample rate, around 125 kB/s. If the host stopped draining for even a moment, async_usb_write_requestWrite() returned false, and ReadLF_realtime() treated that as fatal: it returned PM3_EIO through a goto that also skipped async_usb_write_stop(). The IN endpoint was left busy, every later usb_write() then returned PM3_EIO, and the device went silent until it was physically replugged. A USB bus reset does not clear it -- EP0 keeps working, descriptors read fine, only bulk IN is dead. That is what a long `lf read` looked like from the client: a transfer that stopped early, and then a device that would not answer the next command. A busy endpoint is back pressure, not an error. The sampling loop now waits for the host, up to 100ms, and only gives up if it never comes back. Every exit path closes the async write. The spins inside the USB helpers are bounded and drop the stuck packet instead of turning into an infinite loop, which is the other half of why the device never recovered. Measured against a reader deliberately held at 41 kB/s: before, the stream died at 212238 of 300000 bytes and the unit needed a replug; after, 300000 of 300000 and it still answers. The last packet of a stream was being lost as well. The host's CDC read buffer is two max sized packets, and a bulk IN transfer only completes on a short packet or a full buffer, so a stream ending on a full 64 byte packet was left sitting in a half filled buffer. It showed as exact parity on the packet count: 513 packets requested -> 512 delivered 514 -> 514 515 packets requested -> 514 delivered 516 -> 516 async_usb_write_stop() now always sends a closing packet, the leftover partial bytes when there are any and a zero length packet otherwise, the same way usb_write() already did for its own transfers. On the client side a short transfer is reported instead of being presented as a complete read, the device is told to stop streaming on that path too, and the in place byte counter is repeated with its final value, since the loop only samples it every 10ms and the last line printed was stale. Lastly `lf read` and `lf sniff` cap the sample count to the graph buffer size. Anything past it was streamed and then discarded by getSamplesFromBufEx(), so `-s 1500000` spent about two extra seconds collecting 220000 samples that were thrown away, and printed "Received 1500000 / 1500000 bytes" followed by "Got 1280000 samples" with nothing to explain the gap. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 4 +++ armsrc/lfsampling.c | 33 ++++++++++++++++--- client/src/cmddata.c | 7 ++++ client/src/cmdlf.c | 38 ++++++++++++++++++++- client/src/comms.c | 26 ++++++++------- common_arm/usb/usb_cdc_apis.h | 5 +++ common_arm/usb/usb_cdc_at32.c | 62 +++++++++++++++++++++++++++++++---- common_arm/usb/usb_cdc_at91.c | 54 ++++++++++++++++++++++++++++-- 8 files changed, 203 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cccce2e18..e7fc23674 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,10 @@ All notable changes to this project will be documented in this file. This project uses the changelog in accordance with [keepchangelog](http://keepachangelog.com/). Please use this to write notable changes, which is not the same as git commit log... ## [unreleased][unreleased] +- Fixed `lf read` - a realtime read no longer ends early when the host falls behind, and the device no longer goes silent until it is replugged (@iceman1001) +- Fixed `lf read` - a realtime read ending on a full USB packet now delivers its last 64 bytes, an odd packet count was silently one packet short (@iceman1001) +- Fixed `lf read` - a transfer that stops short says so, instead of being presented as a complete read (@iceman1001) +- Changed `lf read` / `lf sniff` - the sample count is capped to the graph buffer size, the surplus was streamed and then thrown away (@iceman1001) - Added `hf mfdes sim` - SetConfiguration is answered, so a reader can disable FormatPICC, switch the card to a random id, or give it a user defined ATS (@iceman1001) - Added `hf mfdes sim` - ReadSignature is answered for a card image of a generation that has one, and refused the way a genuine EV1 refuses it (@iceman1001) - Added `hf mfdes sim` - GetKeyVersion is answered, so a reader can read the version of any key without knowing the key itself (@iceman1001) diff --git a/armsrc/lfsampling.c b/armsrc/lfsampling.c index 3bcb745ec..c4fb9964a 100644 --- a/armsrc/lfsampling.c +++ b/armsrc/lfsampling.c @@ -460,6 +460,11 @@ uint32_t SampleLF(bool verbose, uint32_t sample_size, bool ledcontrol, bool cota * @param reader_field - true for reading tags, false for sniffing * @return sampling result **/ + +// how long the host is given to collect a streamed USB packet before the link is declared dead? +// ticks run at 1.5MHz, about 100ms +#define LF_REALTIME_USB_TIMEOUT (100 * 1500) + int ReadLF_realtime(bool reader_field, bool cotag, uint32_t sample_limit) { // parameters from config and constants const uint8_t bits_per_sample = config.bits_per_sample; @@ -548,10 +553,21 @@ int ReadLF_realtime(bool reader_field, bool cotag, uint32_t sample_limit) { last_byte = curr_byte; if (samples.total_saved == size_threshold) { - // Request USB transmission and change FIFO bank + + // Request USB transmission and change FIFO bank. + // A busy IN endpoint only means the host has not collected the previous packet yet, if (async_usb_write_requestWrite() == false) { - return_value = PM3_EIO; - goto out; + + uint32_t start_ticks = GetTicks(); + do { + + WDT_HIT(); + if (GetTicksDelta(start_ticks) > LF_REALTIME_USB_TIMEOUT) { + return_value = PM3_EIO; + goto out; + } + + } while (async_usb_write_requestWrite() == false); } // Reset sample @@ -573,9 +589,16 @@ int ReadLF_realtime(bool reader_field, bool cotag, uint32_t sample_limit) { } } - return_value = async_usb_write_stop(); - out: + // Always close the async write down, error paths included. Leaving it open + // strands the IN endpoint with a half filled FIFO, and every later usb_write() fails + { + int stop_value = async_usb_write_stop(); + if (return_value == PM3_SUCCESS) { + return_value = stop_value; + } + } + LED_D_OFF(); // DoAcquisition() end diff --git a/client/src/cmddata.c b/client/src/cmddata.c index 979ecc31b..612acd152 100644 --- a/client/src/cmddata.c +++ b/client/src/cmddata.c @@ -1883,6 +1883,13 @@ int getSamplesFromBufEx(uint8_t *data, size_t sample_num, uint8_t bits_per_sampl size_t max_num = MIN(sample_num, MAX_GRAPH_TRACE_LEN); + if (sample_num > MAX_GRAPH_TRACE_LEN) { + PrintAndLogEx(WARNING, "graph buffer holds " _YELLOW_("%d") " samples, dropping the last " _YELLOW_("%zu") + , MAX_GRAPH_TRACE_LEN + , sample_num - MAX_GRAPH_TRACE_LEN + ); + } + if (bits_per_sample < 8) { if (verbose) PrintAndLogEx(INFO, "Unpacking..."); diff --git a/client/src/cmdlf.c b/client/src/cmdlf.c index ed262118e..97f573b10 100644 --- a/client/src/cmdlf.c +++ b/client/src/cmdlf.c @@ -808,6 +808,7 @@ static int lf_read_internal(bool realtime, bool verbose, uint64_t samples, bool size_t sample_bytes = samples * bits_per_sample; sample_bytes = (sample_bytes / 8) + (sample_bytes % 8 != 0); + const size_t expected_bytes = sample_bytes; // In real-time mode, the LF bitstream should be loaded before receiving raw data. // Otherwise, the first batch of raw data might contain the response of CMD_WTX. @@ -841,6 +842,15 @@ static int lf_read_internal(bool realtime, bool verbose, uint64_t samples, bool } free(realtimeBuf); + + if (sample_bytes < expected_bytes) { + // keep what did arrive, but don't let it look like a complete read + PrintAndLogEx(WARNING, "transfer stopped early, got " _YELLOW_("%zu") " of " _YELLOW_("%zu") " bytes" + , sample_bytes + , expected_bytes + ); + return PM3_ETIMEOUT; + } } else { SendCommandNG(CMD_LF_ACQ_RAW_ADC, (uint8_t *)&payload, sizeof(payload)); PacketResponseNG resp; @@ -895,6 +905,13 @@ int CmdLFRead(const char *Cmd) { bool cm = arg_get_lit(ctx, 3); CLIParserFree(ctx); + // anything past the graph buffer is streamed and then thrown away by + // getSamplesFromBufEx(), so don't spend the transfer time on it + if (samples > MAX_GRAPH_TRACE_LEN) { + PrintAndLogEx(INFO, "Capping to " _YELLOW_("%d") " samples, the graph buffer size", MAX_GRAPH_TRACE_LEN); + samples = MAX_GRAPH_TRACE_LEN; + } + // it should be the result of BigBuf_max_traceLen(), // but IDK how to get it. bool realtime = samples >= g_pm3_capabilities.bigbuf_size; @@ -910,7 +927,9 @@ int CmdLFRead(const char *Cmd) { ret = lf_read_internal(realtime, verbose, samples, false); } while (cm && (kbd_enter_pressed() == false)); - if (ret == PM3_SUCCESS) { + // a truncated transfer still leaves usable samples in the graph buffer, + // so report them. lf_read_internal() has already said it was short. + if (ret == PM3_SUCCESS || ret == PM3_ETIMEOUT) { PrintAndLogEx(SUCCESS, "Got " _YELLOW_("%zu") " samples", g_GraphTraceLen); if (getSignalProperties()->isnoise) { @@ -947,6 +966,7 @@ int lf_sniff(bool realtime, bool verbose, uint64_t samples) { size_t sample_bytes = samples * bits_per_sample; sample_bytes = (sample_bytes / 8) + (sample_bytes % 8 != 0); + const size_t expected_bytes = sample_bytes; // In real-time mode, the LF bitstream should be loaded before receiving raw data. // Otherwise, the first batch of raw data might contain the response of CMD_WTX. @@ -980,6 +1000,15 @@ int lf_sniff(bool realtime, bool verbose, uint64_t samples) { } free(realtimeBuf); + + if (sample_bytes < expected_bytes) { + // keep what did arrive, but don't let it look like a complete read + PrintAndLogEx(WARNING, "transfer stopped early, got " _YELLOW_("%zu") " of " _YELLOW_("%zu") " bytes" + , sample_bytes + , expected_bytes + ); + return PM3_ETIMEOUT; + } } else { SendCommandNG(CMD_LF_SNIFF_RAW_ADC, (uint8_t *)&payload, sizeof(payload)); PacketResponseNG resp; @@ -1027,6 +1056,13 @@ int CmdLFSniff(const char *Cmd) { bool cm = arg_get_lit(ctx, 3); CLIParserFree(ctx); + // anything past the graph buffer is streamed and then thrown away by + // getSamplesFromBufEx(), so don't spend the transfer time on it + if (samples > MAX_GRAPH_TRACE_LEN) { + PrintAndLogEx(INFO, "Capping to " _YELLOW_("%d") " samples, the graph buffer size", MAX_GRAPH_TRACE_LEN); + samples = MAX_GRAPH_TRACE_LEN; + } + // it should be the result of BigBuf_max_traceLen(), // but IDK how to get it. bool realtime = samples >= g_pm3_capabilities.bigbuf_size; diff --git a/client/src/comms.c b/client/src/comms.c index 193c4346d..b2b1a19d4 100644 --- a/client/src/comms.c +++ b/client/src/comms.c @@ -1105,28 +1105,32 @@ size_t WaitForRawDataTimeout(uint8_t *buffer, size_t len, size_t ms_timeout, boo msleep(10); } - if (printed_progress) { - PrintAndLogEx(NORMAL, ""); - } - - if (pos == len && (ms_timeout != (size_t) - 1) && (keep_raw_mode == false)) { - // If ms_timeout != -1, when the desired data is received, tell the arm side - // to stop the current process, and wait for some time to make sure the process - // has been stopped. - // If ms_timeout == -1, the user might not want to break the existing process - // on the arm side. + if ((ms_timeout != (size_t) - 1) && (keep_raw_mode == false)) { SendCommandNG(CMD_BREAK_LOOP, NULL, 0); msleep(ms_timeout); } + // Caller is about to chain another raw read: keep raw-receive mode // on so the comm thread never drops back into normal framed-packet - // parsing in between the two reads. Doing so - even briefly - lets it + // parsing in between the two reads. Doing so lets it // misparse in-flight sample bytes as a PacketResponseNG preamble and // corrupts every USB exchange that follows. if (keep_raw_mode == false) { SetCommunicationReceiveMode(false); } + pos = __atomic_load_n(&comm_raw_pos, __ATOMIC_SEQ_CST); + + if (printed_progress) { + // the loop only samples the counter every 10ms + // Repeat it with the final count before ending the line. + PrintAndLogEx(INPLACE, "Received " _YELLOW_("%zu") " / " _YELLOW_("%zu") " bytes (" _YELLOW_("%zu") "%%)" + , pos + , len + , (pos * 100) / len + ); + PrintAndLogEx(NORMAL, ""); + } return pos; } diff --git a/common_arm/usb/usb_cdc_apis.h b/common_arm/usb/usb_cdc_apis.h index 3f6a5d3de..b0f60ddb1 100644 --- a/common_arm/usb/usb_cdc_apis.h +++ b/common_arm/usb/usb_cdc_apis.h @@ -21,6 +21,10 @@ #include "common.h" +// How many times the async write helpers spin on the IN endpoint before they give up on the host. +// usb_check() stays true for as long as the device is enumerated +#define ASYNC_WRITE_SPIN_LIMIT 0x00020000 + void usb_disable(void); void usb_enable(void); bool usb_check(void); @@ -33,6 +37,7 @@ int usb_write(const uint8_t *data, size_t len); int async_usb_write_start(void); void async_usb_write_pushByte(uint8_t data); bool async_usb_write_requestWrite(void); +void async_usb_write_discard(void); int async_usb_write_stop(void); void usb_update_serial(uint64_t newSerialNumber); diff --git a/common_arm/usb/usb_cdc_at32.c b/common_arm/usb/usb_cdc_at32.c index 2c6b7d4b8..7a68e34f7 100644 --- a/common_arm/usb/usb_cdc_at32.c +++ b/common_arm/usb/usb_cdc_at32.c @@ -1,5 +1,6 @@ #include "pm3_cmd.h" #include "ticks_apis.h" +#include "wdt_apis.h" #include "usb_cdc_apis.h" #include "usb_read_ng.h" #include "usb_cdc_desc.h" @@ -552,10 +553,25 @@ int async_usb_write_start(void) { otg_device_type *dev = OTG_DEVICE(udev->usb_reg); // check usb state - if (!usb_check()) return PM3_EIO; + if (!usb_check()) { + return PM3_EIO; + } // wait for tx end if working... - while (!is_write_completed(ept_in)) if (!usb_check()) return PM3_EIO; + uint32_t spins = 0; + while (!is_write_completed(ept_in)) { + + if (!usb_check()) { + return PM3_EIO; + } + + WDT_HIT(); + + if (++spins > ASYNC_WRITE_SPIN_LIMIT) { + async_usb_write_discard(); + break; + } + } // disable fifo empty irq. dev->diepempmsk &= ~(1 << (USBD_CDC_BULK_IN_EPT & 0x7F)); @@ -617,25 +633,57 @@ bool async_usb_write_requestWrite(void) { return TRUE; } +/** + * Drop a transfer the host refuses to collect. + * Disables the IN endpoint and flushes its tx fifo, otherwise the endpoint + * stays busy forever and every later usb_write() bails out, which looks + * like a dead device. + */ +void async_usb_write_discard(void) { + usbd_ept_in_check_fifo(udev, USBD_CDC_BULK_IN_EPT & 0x7F); + async_write_buf_select = 0; + async_write_index = 0; +} + /** * Stop send and wait finish. * @return SUCCESS if send stop success, otherwise ERROR */ int async_usb_write_stop(void) { otg_eptin_type *ept_in = USB_INEPT(udev->usb_reg, (USBD_CDC_BULK_IN_EPT & 0x7F)); + uint32_t spins = 0; // Wait for the end of transfer - while (!is_write_completed(ept_in)) if (!usb_check()) return PM3_EIO; + while (!is_write_completed(ept_in)) { - // still have data on local buffer, we need send before write stop. - if (async_write_index != 0) { - if (!async_usb_write_requestWrite()) { + if (!usb_check()) { return PM3_EIO; } + + WDT_HIT(); + + if (++spins > ASYNC_WRITE_SPIN_LIMIT) { + async_usb_write_discard(); + return PM3_ETIMEOUT; + } + } + + // Send a final packet. When the local buffer still holds data this flushes it, + // if empty, send a zero length packet. + if (!async_usb_write_requestWrite()) { + return PM3_EIO; } // Wait for the end of fifo flush transfer. - while (!is_write_completed(ept_in)) if (!usb_check()) return PM3_EIO; + spins = 0; + while (!is_write_completed(ept_in)) { + if (!usb_check()) return PM3_EIO; + WDT_HIT(); + if (++spins > ASYNC_WRITE_SPIN_LIMIT) { + async_usb_write_discard(); + return PM3_ETIMEOUT; + } + } return PM3_SUCCESS; } diff --git a/common_arm/usb/usb_cdc_at91.c b/common_arm/usb/usb_cdc_at91.c index 649b9e9cd..df6c859c6 100644 --- a/common_arm/usb/usb_cdc_at91.c +++ b/common_arm/usb/usb_cdc_at91.c @@ -22,6 +22,7 @@ #include "proxmark3_arm.h" #include "usart_defs.h" #include "ticks_apis.h" +#include "wdt_apis.h" #include "usb_read_ng.h" #include "usb_cdc_desc.h" @@ -529,16 +530,44 @@ int async_usb_write_start(void) { return PM3_EIO; } + uint32_t spins = 0; while (pUdp->UDP_CSR[AT91C_EP_IN] & AT91C_UDP_TXPKTRDY) { + if (usb_check() == false) { return PM3_EIO; } + + WDT_HIT(); + + if (++spins > ASYNC_WRITE_SPIN_LIMIT) { + async_usb_write_discard(); + break; + } } isAsyncRequestFinished = false; return PM3_SUCCESS; } +/* + *---------------------------------------------------------------------------- + * \fn async_usb_write_discard + * \brief Drop a packet the host refuses to collect + * + * Without this the endpoint stays busy forever and usb_write() keeps bailing + * out with PM3_EIO, which looks like a dead device. + *---------------------------------------------------------------------------- +*/ +void async_usb_write_discard(void) { + // Clearing TXPKTRDY by hand is not enough on this silicon, the FIFO keeps + // its content. Reset the endpoint the same way the bus reset handler does + // and re-enable it, so the next usb_write() starts from a clean FIFO. + pUdp->UDP_RSTEP |= (1u << AT91C_EP_IN); + pUdp->UDP_RSTEP &= ~(1u << AT91C_EP_IN); + pUdp->UDP_CSR[AT91C_EP_IN] = (AT91C_UDP_EPEDS | AT91C_UDP_EPTYPE_BULK_IN); + isAsyncRequestFinished = true; +} + /* *---------------------------------------------------------------------------- * \fn async_usb_write_pushByte @@ -602,25 +631,46 @@ inline bool async_usb_write_requestWrite(void) { *---------------------------------------------------------------------------- */ int async_usb_write_stop(void) { + + uint32_t spins = 0; + // Wait for the end of transfer while (pUdp->UDP_CSR[AT91C_EP_IN] & AT91C_UDP_TXPKTRDY) { + if (usb_check() == false) { return PM3_EIO; } + + WDT_HIT(); + + if (++spins > ASYNC_WRITE_SPIN_LIMIT) { + async_usb_write_discard(); + return PM3_ETIMEOUT; + } } // clear transmission completed flag UDP_CLEAR_EP_FLAGS(AT91C_EP_IN, AT91C_UDP_TXCOMP); while (pUdp->UDP_CSR[AT91C_EP_IN] & AT91C_UDP_TXCOMP) {}; - // FIFO is not empty, request a write in non-ping-pong mode - if (isAsyncRequestFinished == false) { + // FIFO is not empty, request a write in non-ping-pong mode. + // Must send ZLP to close transfer + { UDP_SET_EP_FLAGS(AT91C_EP_IN, AT91C_UDP_TXPKTRDY); + spins = 0; while (!(pUdp->UDP_CSR[AT91C_EP_IN] & AT91C_UDP_TXCOMP)) { + if (usb_check() == false) { return PM3_EIO; } + + WDT_HIT(); + + if (++spins > ASYNC_WRITE_SPIN_LIMIT) { + async_usb_write_discard(); + return PM3_ETIMEOUT; + } } UDP_CLEAR_EP_FLAGS(AT91C_EP_IN, AT91C_UDP_TXCOMP);