Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kong's log system does not allow you to set fields with dots #11582

Open
1 task done
john-wd opened this issue Sep 15, 2023 · 10 comments
Open
1 task done

Kong's log system does not allow you to set fields with dots #11582

john-wd opened this issue Sep 15, 2023 · 10 comments
Assignees
@hanshuebner
Labels
core/logs plugins/file-log stale/pending revisit Too old ticket. Closed, but we may revisit later. task/feature Requests for new features in Kong

Comments

@john-wd
Copy link

john-wd commented Sep 15, 2023

Is there an existing issue for this?

  • I have searched the existing issues

Kong version ($ kong version)

3.2.2.2

Current Behavior

When I set a custom field to the file-log plugin containing dots, it automatically unpacks the name into nested fields. For example, when I try to set Google's magic trace field logging.googleapis.com/trace, it unpacks as:

{
  logging:
    googleapis:
      com/trace: "my-trace"
}

which does not integrate with google cloud at all.

Expected Behavior

When I set field with dots, I expect it to keep dots

{
  logging.googleapis.com/trace: "my-trace"
}

Steps To Reproduce

  1. Configure file-log setting a custom field containing dots
plugins:
- name: file-log
  config:
    path: "/tmp/file.log"
    custom_fields_by_lua:
      "logging.googleapis.com/trace": "return kong.request.get_header('x-cloud-trace-context')"
  1. Hit any exposed endpoint
  2. See in /tmp/file.log that this field was broken into a nested document

Anything else?

This seems to be the default behavior judging by these lines, but I'd expect to be able to integrate with my cloud provider.

Additional info:

  • Deployed with public helm chart
  • Tag: 3.2.2.2-ubuntu
@StarlightIbuki
Copy link
Contributor

Thanks for reporting this. We can locate the logic handling the dots here.
Maybe we could introduce an escape for the dot. We will discuss this.

@StarlightIbuki StarlightIbuki added task/feature Requests for new features in Kong plugins/file-log core/logs labels Sep 18, 2023
@hanshuebner hanshuebner self-assigned this Sep 18, 2023
@hanshuebner
Copy link
Contributor

Create internal ticket KAG-2601

@github-actions
Copy link
Contributor

github-actions bot commented Oct 5, 2023

This issue is marked as stale because it has been open for 14 days with no activity.

@github-actions github-actions bot added the stale label Oct 5, 2023
@StarlightIbuki StarlightIbuki added the stale/pending revisit Too old ticket. Closed, but we may revisit later. label Oct 20, 2023
@StarlightIbuki
Copy link
Contributor

This should be easy to fix: for example, we could use % to escape . ("%." for . and "%%" for %). But this would be a breaking change. This could fit more in the next major version. I'm trying to confirm this.
Before we have such improvement, maybe we could workaround with a custom plugin.

@mheap
Copy link
Member

mheap commented Nov 2, 2023

Why use % rather than something more standard such as \ ?

@StarlightIbuki
Copy link
Contributor

Why use % rather than something more standard such as \ ?

% is just used to illustrate. To make a choice I will pick a character that is less likely to occur in the name.

@ms2008
Copy link
Contributor

ms2008 commented Nov 2, 2023

The issue is sort of conditionally fixed. That is, when you send a JSON request it works (after this PR #11091), but when you send a form data request it doesn't work because the variable name in the form data contains the . symbol, which is recognized by Kong as nested keys, and then normalize dotted keys in objects which will cause schema violation.

@john-wd
Copy link
Author

john-wd commented Nov 2, 2023

@ms2008 will this fix be a either/or situation in which either all dots in, say custom_fields_by_lia, will be escaped or all of them will break into a nested table?

@ms2008
Copy link
Contributor

ms2008 commented Nov 3, 2023

@catbro666 What about your thoughts?

@catbro666
Copy link
Contributor

@ms2008 This should be a separate issue from #11091 . That one is about validation failure when the key contains dots, while this one is because the dot character has a special meaning in the keys of custom_fields_by_lua but there are no ways to escape it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hanshuebner hanshuebner

Labels
core/logs plugins/file-log stale/pending revisit Too old ticket. Closed, but we may revisit later. task/feature Requests for new features in Kong
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@mheap @hanshuebner @ms2008 @catbro666 @john-wd @StarlightIbuki