[extension/azure_encoding] Add processing for AppService, CDN, FrontDoor and Recommendation logs #44919

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

Fiery-Fenix wants to merge 3 commits into open-telemetry:main from Fiery-Fenix:feat/azureencoding-logs-appservice

Contributor

Fiery-Fenix commented Dec 12, 2025

Description

This is a second part from the set of Azure Resource Log parsing implementations in azure_encoding extension.
This part is bringing this extension functionality in parity with pkg/translator/azurelogs in terms of supported Azure Resource Log Categories (and even a bit more than azurelogs has).

Link to tracking issue

Part of #41725

Testing

Corresponding Unit Test were added.

Documentation

Readme is updated with fields mapping information for supported Azure Resource Log Categories.

Fiery-Fenix requested review from a team and axw as code owners

December 12, 2025 13:40

github-actions bot assigned dehaansa

github-actions bot added the extension/encoding/azureencoding label

github-actions bot requested a review from constanca-m

December 12, 2025 13:41

Fiery-Fenix mentioned this pull request

New component: azureencodingextension #41725

Open

thompson-tomo reviewed

View reviewed changes

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated

    
              | `cacheStatus`             | `azure.cache_status`          | Log Attribute |

              | `errorInfo`               | `exception.type`              | Log Attribute |

              | `endpoint`                | `destination.address` + `destination.port` if it is equal to `hostName` or `network.peer.address` + `network.peer.port` otherwise. If unparsable - `destination.address.original` | Log Attribute |

              | `hostName`                | `destination.address` + `destination.port`. If unparsable - `destination.address.original` | Log Attribute |

Contributor

thompson-tomo Dec 13, 2025 •

edited

Loading

Why is this not

Suggested change

      
            | `hostName`                | `destination.address` + `destination.port`. If unparsable - `destination.address.original` | Log Attribute |
          
            | `hostName`                | `host.name` | Log Attribute |

Contributor Author

Fiery-Fenix Dec 15, 2025

Changed to http.request.header.host as it described in documentation:

The host name in the request from client. If you enable custom domains and have wildcard domain (*.contoso.com), the HostName log field's value is subdomain-from-client-request.contoso.com. If you use the Azure Front Door domain (contoso-123.z01.azurefd.net), the HostName log field's value is contoso-123.z01.azurefd.net

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

thompson-tomo reviewed

View reviewed changes

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

thompson-tomo reviewed

View reviewed changes

Contributor

thompson-tomo left a comment

Have attempted to be consistent with network attributes. Key thing is client/server attributes are 1 pair source/destination is another pair and both pairs shouldn't be used together.

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md

    
              | `requestBytes`            | `http.request.size`           | Log Attribute |

              | `responseBytes`           | `http.response.size`          | Log Attribute |

              | `userAgent`               | `user_agent.original`         | Log Attribute |

              | `clientIp`                | `client.address`              | Log Attribute |

Contributor

thompson-tomo Dec 14, 2025

Suggested change

      
            | `clientIp`                | `client.address`              | Log Attribute |
          
            | `clientIp`                | `source.address`              | Log Attribute |

Contributor

constanca-m Dec 15, 2025

I think source is only meant to be used when it is not clear this is a client, so it is best to stick with client here

Contributor

thompson-tomo Dec 15, 2025 •

edited

Loading

But then we shouldn't have destination.* attributes which was already there. In fact this mapping also already has source.* attributes.

Contributor Author

Fiery-Fenix Dec 15, 2025

Official Microsoft Azure documentation stated that it's "the IP address of the client that made the request", so it should be client.*

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md

    
              | `responseBytes`           | `http.response.size`          | Log Attribute |

              | `userAgent`               | `user_agent.original`         | Log Attribute |

              | `clientIp`                | `client.address`              | Log Attribute |

              | `clientPort`              | `client.port`                 | Log Attribute |

Contributor

thompson-tomo Dec 14, 2025

Suggested change

      
            | `clientPort`              | `client.port`                 | Log Attribute |
          
            | `clientPort`              | `source.port`                 | Log Attribute |

Contributor

constanca-m Dec 15, 2025

#44919 (comment)

Contributor Author

Fiery-Fenix Dec 15, 2025

Official Microsoft Azure documentation stated that it's "The IP port of the client that made the request", so it should be client.*

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated

    
              | Azure "properties" Field  | OpenTelemetry                     | OpenTelemetry Scope |

              |---------------------------|-----------------------------------|---------------------|

              | `clientIP`                | `client.address`                  | Log Attribute |

Contributor

thompson-tomo Dec 14, 2025

Suggested change

      
            | `clientIP`                | `client.address`                  | Log Attribute |
          
            | `clientIP`                | `source.address`                  | Log Attribute |

Contributor

constanca-m Dec 15, 2025

#44919 (comment)

Contributor

thompson-tomo Dec 15, 2025

For me Firewall rules don't fit into the client/server representation given that firewall rules are usually defined using the notion of source/destination.

Contributor Author

Fiery-Fenix Dec 15, 2025

Official Microsoft Azure documentation stated that it's "The IP address of the client that made the request", so it should be client.*

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated

    
              | Azure "properties" Field  | OpenTelemetry                     | OpenTelemetry Scope |

              |---------------------------|-----------------------------------|---------------------|

              | `clientIP`                | `client.address`                  | Log Attribute |

              | `clientPort`              | `client.port`                     | Log Attribute |

Contributor

thompson-tomo Dec 14, 2025

Suggested change

      
            | `clientPort`              | `client.port`                     | Log Attribute |
          
            | `clientPort`              | `source.port`                     | Log Attribute |

Contributor

constanca-m Dec 15, 2025

#44919 (comment)

Contributor Author

Fiery-Fenix Dec 15, 2025

Official Microsoft Azure documentation stated that it's "The IP port of the client that made the request.", so it should be client.*

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md

    
              | `requestBytes`            | `http.request.size`           | Log Attribute |

              | `responseBytes`           | `http.response.size`          | Log Attribute |

              | `userAgent`               | `user_agent.original`         | Log Attribute |

              | `clientIp`                | `client.address`              | Log Attribute |

Contributor

thompson-tomo Dec 14, 2025

Suggested change

      
            | `clientIp`                | `client.address`              | Log Attribute |
          
            | `clientIp`                | `source.address`              | Log Attribute |

Contributor Author

Fiery-Fenix Dec 15, 2025

Official Microsoft Azure documentation stated that it's "The IP address of the client that made the original request", so it should be client.*

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md

    
              | `responseBytes`           | `http.response.size`          | Log Attribute |

              | `userAgent`               | `user_agent.original`         | Log Attribute |

              | `clientIp`                | `client.address`              | Log Attribute |

              | `clientPort`              | `client.port`                 | Log Attribute |

Contributor

thompson-tomo Dec 14, 2025

Suggested change

      
            | `clientPort`              | `client.port`                 | Log Attribute |
          
            | `clientPort`              | `source.port`                 | Log Attribute |

Contributor Author

Fiery-Fenix Dec 15, 2025

Official Microsoft Azure documentation stated that it's "The IP port of the client that made the request.", so it should be client.*

extension/encoding/azureencodingextension/internal/unmarshaler/logs/README.md Outdated Show resolved Hide resolved

Contributor Author

Fiery-Fenix commented Dec 14, 2025 •

edited

Loading

Have attempted to be consistent with network attributes. Key thing is client/server attributes are 1 pair source/destination is another pair and both pairs shouldn't be used together.

I don't think that using source/destination everywhere is the correct approach.
According to SemConv documentation source/destination is used for low-level network communication and/or when there is no client/server relationship. This might be valid for firewall logs (still not sure), but definitely not valid for HTTP access logs (FronDoor, CDN, AppService access logs).
I think we should use client.*/server.* attributes at least for HTTP Access Logs here, because it logs for high-level (OSI level 7) communication protocol and definitely knows which side initiates the connection.

Contributor

thompson-tomo commented Dec 14, 2025

So the access logs are a hard one to place and they could be client/server however the existing proposal was already using source & destination namespace hence I consolidated on that.

axw reviewed

View reviewed changes

.../azureencodingextension/internal/unmarshaler/logs/testdata/appservice/http-log_expected.yaml Outdated Show resolved Hide resolved

...oding/azureencodingextension/internal/unmarshaler/logs/testdata/appservice/platform-log.json

Comment on lines +14 to +18

    
                          "EventStampType": "Stamp",

                          "EventPrimaryStampName": "waws-prod-blu-479",

                          "EventStampName": "waws-prod-blu-479",

                          "Host": "lw0sdlwk0005XR",

                          "EventIpAddress": "10.50.0.34"

Contributor

axw Dec 15, 2025

Is it intentional that these details are dropped?

Contributor Author

Fiery-Fenix Dec 15, 2025

This field is not described in documentation for this category, so I'm not sure how to correctly map it to OpenTelemetry attributes as the meaning of this value is unclear

.../azureencodingextension/internal/unmarshaler/logs/testdata/azurecdn/access-log_expected.yaml Outdated Show resolved Hide resolved

.../azureencodingextension/internal/unmarshaler/logs/testdata/azurecdn/access-log_expected.yaml Outdated Show resolved Hide resolved

Contributor Author

Fiery-Fenix commented Dec 15, 2025

I have uploaded all the changes discussed here, no more mixtures of client.* and destination.*, updated mapping documentation and some cleanup based on Microsoft Azure documentation re-check.

As for client.* vs source.* - I have double checked official Microsoft Azure documentation for all categories and decided to use it as a source of truth.
So client.* vs source.* usage rules are simple:

If official Microsoft Azure documentation explicitly states that it's log for client-server communication and it has request initiator field - use client.*/server.*. For example from Azure CDN documentation: "ClientIp | The IP address of the client that made the request. ..." - documentation explicitly stated that we have client that initiate request and we have a field for it's address, decision - use client.address.
Otherwise - use source.*/destination.*

constanca-m reviewed

View reviewed changes

extension/encoding/azureencodingextension/internal/unmarshaler/logs/category.go

Comment on lines +83 to +85

    
              	// OpenTelemetry attribute name for Azure HTTP Request Duration,

              	// from `durationMs` field in Azure Log Record

              	attributeAzureRequestDuration = "azure.request.duration"

Contributor

constanca-m Dec 15, 2025

If it is client, you should http.client.request.duration, if it is server, you should use http.server.request.duration. See https://opentelemetry.io/docs/specs/semconv/dotnet/dotnet-http-metrics/

Contributor Author

Fiery-Fenix Dec 15, 2025

Unfortunately, type of value is conflicting with SemConv definition - SemConv states that it should be Histogram while our value is a single number

Fiery-Fenix closed this

Fiery-Fenix reopened this

Fiery-Fenix added 2 commits

December 17, 2025 17:10


          [extension/azure_encoding] Add processing for AppService, CDN, FrontD…

2a23f77

…oor and Recommendation logs


          Update attributes naming after review

5b506db

Fiery-Fenix force-pushed the feat/azureencoding-logs-appservice branch from c5a64d1 to 5b506db Compare

December 17, 2025 15:10


          Update tests to use string representation of semconv

de54bca

Contributor Author

Fiery-Fenix commented Dec 17, 2025

Rebased and updated tests to fulfill new lint requirements (SemConv attribute names in tests should be a string representation now)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extension/encoding/azureencoding