Forums - Problem: OTA Bootloader in EEPROM Being Corrupted

6 posts / 0 new
Last post
Problem: OTA Bootloader in EEPROM Being Corrupted
sunlong
Join Date: 27 Nov 19
Posts: 8
Posted: Mon, 2020-06-22 00:18

Hi there,

I m struggling to investigate a weird problem.

Problem:

In short, the problem is : the OTA bootloader part in EEPROM is corrupted.

The device can't boot up and doesn't repond to GPIO interrupt. LED keesp lighting on all the time which should blink after boot up.

When I try to read CS configurations of bootloader using CsConfig.exe, it reported  like this:

 

What I tried:

I compared image file dump from devices being investigaed with good image file, seems there are some continious difference from 0x000026 to 0x000058:

 

Information about EEPROM map:

mostly followed examples from CSR_uEnergy_SDK-2.6.1.7

Application NVM settings in csr101x_A05.keyr:

&nvm_start_address = 4100 // Default value (in hex) for EEPROM

&nvm_size = 40 // Default value(number of words)

 

Will be much appreciate if someone could share some thoughts on this.

 

B.R.

 

  • Up0
  • Down0
sunlong
Join Date: 27 Nov 19
Posts: 8
Posted: Mon, 2020-06-22 00:43

Here I uploaded the screenshots which is not loaded in my last post:

 

  • Up0
  • Down0
abhghos
Join Date: 27 Aug 18
Posts: 18
Posted: Mon, 2020-06-22 02:47

This indeed is a wierd issue. Looks like some section of the EEPROM has 0xbbbb over writted by some rouge process.

What is the occuerence of this issue? How frequently is this seen? Can this be reproduced?

What is the version of the OTA? v6 or v7?

The latest available SDK for CSR1010 is 2.6.3.12. Recommend to use the same.

  • Up0
  • Down0
sunlong
Join Date: 27 Nov 19
Posts: 8
Posted: Mon, 2020-06-22 03:30

Hi abhgos,

Thanks for paying attention here.

 

Q : What is the occuerence of this issue? How frequently is this seen? Can this be reproduced?

A :  Till now I just found 2 bad samples of 1000 pcs devices. I am still working to find out how to reproduce this phenomenon.

 

Q : What is the version of the OTA?

A : It's V7.

 

Q : The latest available SDK for CSR1010 is 2.6.3.12. Recommend to use the same.

A : Thanks for this advice. Will apply latest SDK to my legacy project later. But investigation with current SDK and project comes first.

 

In fact, the application has some chanes to manipulate NVM, such as store bonding information and some other user data.

Is it possible the NVM interface have some bugs to cause such NVM damage in bootloader code sector?

Or would this be related to power supply chain?

 

B.R.

  • Up0
  • Down0
abhghos
Join Date: 27 Aug 18
Posts: 18
Posted: Mon, 2020-06-22 05:12

Hi,

As you can understand, its really difficult to figure out a actual root cause here. Lot of testing is needed to conclude.

The EEPROM read write is realtively stable and I don't think that a power supply can cause this. Is the memory corrupted in the same place in both the faulty devices? If yes, this would strongly indicate some bug with the firmware.

However, it will be nice to add some protection in the situation where the Application does NVM Write calls. The write address should fall within some specified limits.

These are the only thoughts coming to my mind now.

  • Up0
  • Down0
sunlong
Join Date: 27 Nov 19
Posts: 8
Posted: Mon, 2020-06-22 16:20

Thank you abhghos,

Q : Is the memory corrupted in the same place in both the faulty devices? If yes, this would strongly indicate some bug with the firmware.

A : yes, it's corrupted in the same place in both faulty devices.

 

it will be nice to add some protection in the situation where the Application does NVM Write calls. The write address should fall within some specified limits.

Yes, this makes sense.  but seems the API NvmWrite only takes offset as NVM address. 

If it takes an big offset out of NVM size, it only should corrupt Application 1 or Application 2 area. More likely, if offset out of NVM size, it should return an error message like "nvm_status_invalid_offset".

extern sys_status NvmWrite(const uint16* buffer, uint16 length, uint16 offset);

NVM memory map:

0x0000 CSR OTA Update bootloader

0x4000 CSR OTA Update Shared Data
0x4100 Application NVM Store
0x4200 Application 1
0xa100 Application 2
0x10000 End of address space

 

BTW, I post the NVM read/write wrapper code in my project:

extern void Nvm_Read(uint16* buffer, uint16 length, uint16 offset)
{
    sys_status result;
 
    /* Read from NVM. Firmware re-enables the NVM if it is disabled */
    result = NvmRead(buffer, length, offset);
 
    /* Disable NVM to save power after read operation */
    Nvm_Disable();
 
    /* Report panic is NVM read is not successful */
    if(sys_status_success != result)
    {
        ReportPanic(app_panic_nvm_read);
    }

 

}
 
extern void Nvm_Write(uint16* buffer, uint16 length, uint16 offset)
{
    sys_status result;
    
    /* Write to NVM. Firmware re-enables the NVM if it is disabled */
    result = NvmWrite(buffer, length, offset);
 
    /* Disable NVM to save power after write operation */
    Nvm_Disable();
 
    /* If NvmWrite was a success, return */
    if(sys_status_success == result)
    {
        /* Write was successful. */
        return;
    }
#ifdef NVM_TYPE_FLASH
    else if(nvm_status_needs_erase == result)
    {
        /* The application already has a copy of NVM data in its variables,
         * so we can erase the NVM 
         */
        Nvm_Erase();
 
        /* Write back the NVM data. 
         * Please note that the following function writes data into NVM and 
         * should not fail. 
         */
         WriteApplicationAndServiceDataToNVM();
    }
#endif /* NVM_TYPE_FLASH */
    else
    {
        /* Irrecoverable error. Reset the chip. */
        ReportPanic(app_panic_nvm_write);
    }
}

 

  • Up0
  • Down0
or Register

Opinions expressed in the content posted here are the personal opinions of the original authors, and do not necessarily reflect those of Qualcomm Incorporated or its subsidiaries (“Qualcomm”). The content is provided for informational purposes only and is not meant to be an endorsement or representation by Qualcomm or any other party. This site may also provide links or references to non-Qualcomm sites and resources. Qualcomm makes no representations, warranties, or other commitments whatsoever about any non-Qualcomm sites or third-party resources that may be referenced, accessible from, or linked to this site.