Feature request: LIN bus *.LDF to *.DBC format

[MatinF]

Hi Eduard,

Would it be possible to support loading a LIN bus *.LDF file into canmatrix - with the use case of e.g. creating a DBC file from the loaded LDF file?

This would e.g. allow tools like asammdf that rely on canmatrix to let users load *.LDF files directly to decode LIN bus data - or to alternatively use canmatrix to first create a DBC file from their LDF files.

We would be able to provide sample LIN bus data in the MDF4 format as well as LDF files for this purpose via e-mail. Further, we would be happy to work with you to test and evaluate the code.

Let me know if this is something you'd be open to.

There may be some option to leverage source code from e.g. this module:
https://github.com/uCAN-LIN/LinUSBConverter

[danielhrisca]

It would be great to load the LDF files as canmatrix.Matrix objects as well

[reymor]

@MatinF @ebroecker

Hello,

I'm open to contribute to this new feature. Let me know how I could help you.

[ebroecker]

using ldfparser (https://github.com/c4deszes/ldfparser) looks very promising to me.

Gonna hack something thus we have a base to discuss.

[ebroecker]

very simply support in:
https://github.com/ebroecker/canmatrix/tree/iss531_import_ldf

https://github.com/ebroecker/canmatrix/blob/iss531_import_ldf/src/canmatrix/formats/ldf.py

maybe you can have a look, ...

[MatinF]

Awesome Eduard. Daniel if you get the chance to test adding this to asammdf I can do some tests

[MatinF]

Hi Eduard, would you be able to include this functionality in an upcoming release? Then it may be possible to include it in e.g. asammdf and our Python API.

[ebroecker]

Does it fullfill your needs?
Did you test?

If it works for your needs certainly I'll add it to a future release

[danielhrisca]

I'm getting this error when I try to load a LDF file:

File "D:\02__PythonWorkspace\_venv\lib\site-packages\canmatrix\formats\__init__.py", line 96, in load
    dbs[key] = module_instance.load(file_object, **options)  # type: ignore
  File "D:\02__PythonWorkspace\_venv\lib\site-packages\canmatrix\formats\ldf.py", line 7, in load
    ldf = ldfparser.parseLDF(path=f.name)  # using f.name is not nice, but works
AttributeError: '_io.BytesIO' object has no attribute 'name'

[danielhrisca]

I've managed to load the LDF file only by using canmatrix.formats.loadp instead of canmatrix.formats.loads

Here are the first missing items:

• conversion rules are not loaded
• I don't think that there is an explicit signed/unsigned declaration in the ldf so this would have to be deduced from the conversion definition (there it has the min and max raw range)

[MatinF]

It should also be noted perhaps that some LIN messages contain signals that have differing decoding rules depending on the physical value. For example the below message:

  Enc_BatteryCurrent {
    physical_value, 1, 66, 10, -1230, "Ampere" ;
    physical_value, 67, 4765, 0.1, -576.6, "Ampere" ;
    physical_value, 4766, 12765, 0.01, -147.66, "Ampere" ;
    physical_value, 12766, 52765, 0.001, -32.766, "Ampere" ;
    physical_value, 52766, 60765, 0.01, -507.66, "Ampere" ;
    physical_value, 60766, 65465, 0.1, -5976.6, "Ampere" ;
    physical_value, 65466, 65532, 10, -654090, "Ampere" ;
    logical_value, 0, "IBatt_lowerlimit" ;
    logical_value, 65533, "IBatt_upperlimit" ;
    logical_value, 65534, "BatteryCurrent_Error" ;
    logical_value, 65535, "LIN_INIT" ;
  }

In such cases, the decoding rule logic cannot be fully handled within the DBC framework. My recommendation for now would be to simply ignore such messages if possible.

[ebroecker]

Thanks for the feedback, gonna work on it

[MatinF]

In regards to comparing LDF vs DBC, you might also find the below article section useful Eduard:
https://www.csselectronics.com/screen/page/lin-bus-protocol-intro-basics/language/en#lin-description-file

Martin

[ebroecker]

did someone have a chance to check acdd5d3 ?

[MatinF]

@danielhrisca not sure if this is the version you included in the asammdf dev version I tested

[danielhrisca]

@ebroecker the new commit throws this exception in calc_max

File "D:\02__PythonWorkspace\_venv\lib\site-packages\canmatrix\canmatrix.py", line 395, in calc_max
    return self.offset + (self.float_factory(rawMax) * self.factor)
TypeError: unsupported operand type(s) for *: 'decimal.Decimal' and 'float'

I think all LDF signals should be parsed as unsigned. @MatinF please confirm if this is true.

Another problem is that there should have a way to store all the conversions/decoding rules. @ebroecker do you think this could be feasible?

[ebroecker]

I'll close this issue, please open new issues for problems with current implementation

[MatinF]

Hi Edward,

I would like to test this out to e.g. see if I can load attached LDF file and export it to DBC. I tried using below script, but I get this error. I tested with the latest release build of canmatrix:

C:\Users\marti\Downloads\LIN_LDF_EXAMPLE>python testy.py LIN_LDF_EXAMPLE.ldf test.dbc
ldf is not supported
xls is not supported
xlsx is not supported
Loading LDF file: LIN_LDF_EXAMPLE.ldf
Error during conversion: Failed to parse the LDF file.

Here is the script:

import canmatrix.formats
import sys

def convert_ldf_to_dbc(ldf_file: str, dbc_file: str):
    """
    Converts an LDF (LIN Description File) to a DBC (CAN Database File).
    
    Parameters:
        ldf_file (str): Path to the input LDF file.
        dbc_file (str): Path to the output DBC file.
    """
    try:
        # Load the LDF file
        print(f"Loading LDF file: {ldf_file}")
        matrix = canmatrix.formats.loadp(ldf_file)
        
        # Check if the loading was successful
        if matrix is None:
            raise ValueError("Failed to parse the LDF file.")
        
        print(f"Successfully loaded LDF file: {ldf_file}")
        
        # Export the matrix to a DBC file
        print(f"Exporting to DBC file: {dbc_file}")
        canmatrix.formats.dump(matrix, dbc_file)
        
        print(f"Successfully exported to DBC file: {dbc_file}")
    except Exception as e:
        print(f"Error during conversion: {e}")

if __name__ == "__main__":
    # Example usage: python convert_ldf_to_dbc.py input.ldf output.dbc
    if len(sys.argv) != 3:
        print("Usage: python convert_ldf_to_dbc.py <input_ldf_file> <output_dbc_file>")
        sys.exit(1)
    
    input_ldf = sys.argv[1]
    output_dbc = sys.argv[2]
    convert_ldf_to_dbc(input_ldf, output_dbc)

The file I am trying to load is attached. Any idea what the issue may be?

LIN_LDF_EXAMPLE.zip

[reymor]

Hello @MatinF

I think that you need to install ldfparser package

[MatinF]

Thanks - I tried installing ldfparser, but now I get this error:

C:\Users\marti\Downloads\LIN_LDF_EXAMPLE>python testy.py IBS_200X_P_24V.ldf test2.dbc
xls is not supported
xlsx is not supported
Loading LDF file: IBS_200X_P_24V.ldf
Successfully loaded LDF file: IBS_200X_P_24V.ldf
Exporting to DBC file: test2.dbc
Error during conversion: dump() missing 1 required positional argument: 'export_type'
```

[MatinF]

I tried updating the dump() function to take a 3rd input, "dbc", which results in below:

C:\Users\marti\Downloads\LIN_LDF_EXAMPLE>python testy.py IBS_200X_P_24V.ldf test2.dbc
xls is not supported
xlsx is not supported
Loading LDF file: IBS_200X_P_24V.ldf
Successfully loaded LDF file: IBS_200X_P_24V.ldf
Exporting to DBC file: test2.dbc
Successfully exported to DBC file: test2.dbc

However, the file is not exported to the expected location that I provide. It seems to load correctly if I print the matrix object, so I guess the problem is related to my use of canmatrix.formats.dump.

[MatinF]

Sorry, now I got it working after using dumpp instead of dump. I.e. the below script works as intended:

import canmatrix.formats
import sys

def convert_ldf_to_dbc(ldf_file: str, dbc_file: str):
    """
    Converts an LDF (LIN Description File) to a DBC (CAN Database File).
    
    Parameters:
        ldf_file (str): Path to the input LDF file.
        dbc_file (str): Path to the output DBC file.
    """
    try:
        # Load the LDF file
        print(f"Loading LDF file: {ldf_file}")
        matrix = canmatrix.formats.loadp(ldf_file)
        
        print("matrix",matrix)
        # Check if the loading was successful
        if matrix is None:
            raise ValueError("Failed to parse the LDF file.")
        
        print(f"Successfully loaded LDF file: {ldf_file}")
        
        # Export the matrix to a DBC file
        print(f"Exporting to DBC file: {dbc_file}")
        dbc = canmatrix.formats.dumpp(matrix, dbc_file, "dbc")
        
        print(f"Successfully exported to DBC file: {dbc_file}")
    except Exception as e:
        print(f"Error during conversion: {e}")

if __name__ == "__main__":
    # Example usage: python convert_ldf_to_dbc.py input.ldf output.dbc
    if len(sys.argv) != 3:
        print("Usage: python convert_ldf_to_dbc.py <input_ldf_file> <output_dbc_file>")
        sys.exit(1)
    
    input_ldf = sys.argv[1]
    output_dbc = sys.argv[2]
    convert_ldf_to_dbc(input_ldf, output_dbc)

One observation in this regard:

Some LIN signals like BatteryCurrent in the example file have multiple physical value ranges as per below in the MDF:

  Enc_BatteryCurrent {
    physical_value, 1, 66, 10, -1230, "Ampere" ;
    physical_value, 67, 4765, 0.1, -576.6, "Ampere" ;
    physical_value, 4766, 12765, 0.01, -147.66, "Ampere" ;
    physical_value, 12766, 52765, 0.001, -32.766, "Ampere" ;
    physical_value, 52766, 60765, 0.01, -507.66, "Ampere" ; 
    physical_value, 60766, 65465, 0.1, -5976.6, "Ampere" ;
    physical_value, 65466, 65532, 10, -654090, "Ampere" ;
    logical_value, 0, "IBatt_lowerlimit" ;
    logical_value, 65533, "IBatt_upperlimit" ;
    logical_value, 65534, "BatteryCurrent_Error" ;
    logical_value, 65535, "LIN_INIT" ;
  }

In the DBC file context, this could be handled via extended multiplexing as showcased below:

Would this be possible to introduce in the parsing of LDF files so that multiple physical value ranges get translated automatically into extended multiplexing ranges?

Currently it looks like the logic is to include the 'last' physical entry. This can be a starting point for the purpose of speeding up the creation of a DBC file - but it would make it non-usable for loading LDF files in e.g. the asammdf GUI for decoding LIN bus log files (as many LIN signals have multiple physical value ranges).

[reymor]

I think that it is because there is no 1to1 conversion between dbc and ldf.

I understand that multiple entries for physical value is not something possible in dbc file format. However,

Would this be possible to introduce in the parsing of LDF files so that multiple physical value ranges get translated automatically into extended multiplexing ranges?

I think that it is possible

So, here let me understand what you mean:

  Enc_BatteryCurrent {
    physical_value, 1, 66, 10, -1230, "Ampere" ;
    physical_value, 67, 4765, 0.1, -576.6, "Ampere" ;
    physical_value, 4766, 12765, 0.01, -147.66, "Ampere" ;
    physical_value, 12766, 52765, 0.001, -32.766, "Ampere" ;
    physical_value, 52766, 60765, 0.01, -507.66, "Ampere" ;
    physical_value, 60766, 65465, 0.1, -5976.6, "Ampere" ;
    physical_value, 65466, 65532, 10, -654090, "Ampere" ;
    logical_value, 0, "IBatt_lowerlimit" ;
    logical_value, 65533, "IBatt_upperlimit" ;
    logical_value, 65534, "BatteryCurrent_Error" ;
    logical_value, 65535, "LIN_INIT" ;
  }

How do you use the extended multiplexing here?

[MatinF]

I believe a 1:1 mapping can be achieved for most LIN signals, with the primary challenge being the cases where a single LIN signal has multiple physical value representations (as in the example I provided). My suggestion in this case would be to consider if this could be handled via extended multiplexing. I illustrate how this would work for the specific example of BatteryCurrent via the picture I shared from Kvaser's DBC editor. Here I use the 16 bits reflecting the BatteryCurrent range and I essentially use that as a multiplexor to modify the interpretation of the same 16 bit range for the purpose of calculating a BatteryCurrent1, BatteryCurrent2, ... signal. Essentially adding a suffix index for each unique physical_value entry of the signal.

The resulting DBC works as intended in the sense that it will correctly DBC decode the data using the scaling/offset specified for the range defined in the multiplexing.

Let me know if it makes more sense? I am not sure if it becomes too complex to create this form of conversion in canmatrix, though. Maybe a starting point could be that canmatrix is able to load the LDF and correctly map the various physical_value ranges correctly for the internal database object somehow - I am not sure if that is done currently. The hope would be to allow tools like the asammdf GUI to then properly load LDF files and use it for decoding MF4 files that contain LIN bus data out-the-box. If this could be achieved, it would to us be more relevant than e.g. being able to export a DBC equivalent format.

[MatinF]

Looking at the internal format of the canmatrix database, I guess the fundamental issue is that - similar to DBC files - the internal format only expects a single scale/offset value per signal (although it is able to also incorporate the 'value table' provided by the logical values in the LDF). For example, the aforementioned BatteryCurrent signal seems to look as below in the internal format:

Frame(
    name="IBS_DataAcquisition",
    arbitration_id=ArbitrationId(
        id=33,
        extended=False
    ),
    size=8,
    transmitters=["IBS"],
    is_complex_multiplexed=False,
    is_fd=False,
    comment="",
    signals=[
        Signal(
            name="BatteryCurrent",
            start_bit=0,
            size=16,
            is_little_endian=True,
            is_signed=False,
            offset=-654090.0,
            factor=10.0,
            unit="Ampere",
            receivers=["SG"],
            comment=None,
            multiplex=None,
            mux_value=None,
            is_float=False,
            is_ascii=False,
            type_label="",
            enumeration=None,
            comments={},
            attributes={},
            values={
                0: "IBatt_lowerlimit",
                65533: "IBatt_upperlimit",
                65534: "BatteryCurrent_Error",
                65535: "LIN_INIT"
            },
            mux_val_grp=[],
            muxer_for_signal=None,
            calc_min_for_none=True,
            calc_max_for_none=True,
            cycle_time=0,
            initial_value=65535,
            min=65466,
            max=65532
        ),
        ...
    ]
)

As evident, the factor/offset corresponds to the last physical_value entry in the signal description of the LDF:

offset=-654090.0,
factor=10.0,

I guess that in order to properly resolve this, a rather fundamental structural change would be required in the canmatrix internal database format to handle multiple possible physical value entries per signal. One option might be to indeed leverage extended multiplexing as described above, as this seems to be supported by canmatrix from what I can tell. This would have the benefit that it would allow a direct export to DBC files as well. But it has the downside that a single signal, BatteryCurrent would have to be mapped to multiple signals, e.g. BatteryCurrent1, BatteryCurrent2, ... to reflect the full range of physical values described in the LDF.

Edit: For context, this is how the encoding section for the BatteryCurrent signal looks from the ldfparser library if I load the LDF and export the contents to JSON:

{
      "name": "Enc_BatteryCurrent",
      "values": [
        {
          "type": "physical",
          "min": 1,
          "max": 66,
          "scale": 10.0,
          "offset": -1230.0,
          "unit": "Ampere"
        },
        {
          "type": "physical",
          "min": 67,
          "max": 4765,
          "scale": 0.1,
          "offset": -576.6,
          "unit": "Ampere"
        },
        {
          "type": "physical",
          "min": 4766,
          "max": 12765,
          "scale": 0.01,
          "offset": -147.66,
          "unit": "Ampere"
        },
        {
          "type": "physical",
          "min": 12766,
          "max": 52765,
          "scale": 0.001,
          "offset": -32.766,
          "unit": "Ampere"
        },
        {
          "type": "physical",
          "min": 52766,
          "max": 60765,
          "scale": 0.01,
          "offset": -507.66,
          "unit": "Ampere"
        },
        {
          "type": "physical",
          "min": 60766,
          "max": 65465,
          "scale": 0.1,
          "offset": -5976.6,
          "unit": "Ampere"
        },
        {
          "type": "physical",
          "min": 65466,
          "max": 65532,
          "scale": 10.0,
          "offset": -654090.0,
          "unit": "Ampere"
        },
        {
          "type": "logical",
          "value": 0,
          "text": "IBatt_lowerlimit"
        },
        {
          "type": "logical",
          "value": 65533,
          "text": "IBatt_upperlimit"
        },
        {
          "type": "logical",
          "value": 65534,
          "text": "BatteryCurrent_Error"
        },
        {
          "type": "logical",
          "value": 65535,
          "text": "LIN_INIT"
        }
      ]
    },

@ebroecker would be interested in your view on this and the feasibility of introducing this aspect of the LDF format into the canmatrix database format.

[ebroecker]

For now we have only valuetables (1:1 Value to Label mapping) and scaling with factor and offset.
Anyway in LDF but also in ARXML it is possible to have different scales for different ranges.

I think it could be the right thing to do to implement something that supports these more flexible raw to physical mappings.

[ebroecker]

I added a simple change ,

#823

feel free to discuss

[MatinF]

Awesome Eduard, appreciate it! I think this would be a really useful addition to enable the decoding of LIN data in various tools with LDF files.

@danielhrisca If you get the chance to try and incorporate this in a dev build, I'd be happy to do some testing on our sample data to verify if this works as intended.

[MatinF]

@ebroecker Will it be possible to merge this change into the master branch and release this as an update to the canmatrix repository?