Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: dbt-ext does not seem to respect Meltano logging configuration #59

Open
menzenski opened this issue Mar 8, 2024 · 0 comments
Open

[Bug]: dbt-ext does not seem to respect Meltano logging configuration #59

menzenski opened this issue Mar 8, 2024 · 0 comments

Comments

@menzenski
Copy link

Package Version

d8582ff

Python Version

NA

Operating System

macOS 13.6

Description

I'm using Python 3.11.3 (wasn't an option in the dropdown in this issue form).

I'm using the following plugin config in meltano.yml:

plugins:
  utilities:
    - name: dbt-postgres
      variant: dbt-labs
      pip_url: dbt-core==1.5.2 dbt-postgres==1.5.2 git https://github.com/meltano/dbt-ext.git@d8582ff16828512c4d7d3a25fe46b53df2483da2
      config:
        skip_pre_invoke: true

And this Meltano logging configuration in logging.yaml:

version: 1
disable_existing_loggers: false
formatters:
  json:
    (): meltano.core.logging.json_formatter
handlers:
  console:
    class: logging.StreamHandler
    formatter: json
    stream: ext://sys.stderr
root:
  level: INFO
  propagate: 'yes'
  handlers:
    - console

I have the environment variables MELTANO_CLI_LOG_CONFIG=logging.yaml, MELTANO_LOG_LEVEL=info, and DBT_LOG_FORMAT=json set.

When I run a dbt command via the meltano invoke wrapper command, I observe the following: the "core" meltano logs are in JSON format, and the dbt logs themselves are in JSON format, but there are some log lines not in JSON - I believe that these are the log lines emitted by the dbt-ext package.

Here's an example:

$ meltano invoke dbt-postgres compile
{"event": "Environment 'dev' is active", "level": "info", "timestamp": "2024-03-08T12:40:40.112179Z"}
Extension executing `dbt compile`...
{"data": {"log_version": 3, "version": "=1.5.2"}, "info": {"category": "", "code": "A001", "extra": {}, "invocation_id": "43f312f3-af1c-4925-966c-c0c231daa041", "level": "info", "msg": "Running with dbt=1.5.2", "name": "MainReportVersion", "pid": 2968, "thread": "MainThread", "ts": "2024-03-08T12:40:51.302737Z"}}
{"data": {"deprecated_path": "target-path"}, "info": {"category": "", "code": "D011", "extra": {}, "invocation_id": "43f312f3-af1c-4925-966c-c0c231daa041", "level": "warn", "msg": "[\u001b[33mWARNING\u001b[0m]: Deprecated functionality\nThe `target-path` config in `dbt_project.yml` has been deprecated, and will no\nlonger be supported in a future version of dbt-core. If you wish to write dbt\nartifacts to a custom directory, please use the --target-path CLI flag or\nDBT_TARGET_PATH env var instead.", "name": "ConfigTargetPathDeprecation", "pid": 2968, "thread": "MainThread", "ts": "2024-03-08T12:40:51.356799Z"}}
{"data": {"adapter_name": "postgres", "adapter_version": "=1.5.2"}, "info": {"category": "", "code": "E034", "extra": {}, "invocation_id": "43f312f3-af1c-4925-966c-c0c231daa041", "level": "info", "msg": "Registered adapter: postgres=1.5.2", "name": "AdapterRegistered", "pid": 2968, "thread": "MainThread", "ts": "2024-03-08T12:40:51.378662Z"}}
{"data": {"stat_line": "124 models, 492 tests, 0 snapshots, 0 analyses, 1189 macros, 2 operations, 6 seed files, 338 sources, 1 exposure, 0 metrics, 0 groups"}, "info": {"category": "", "code": "W006", "extra": {}, "invocation_id": "43f312f3-af1c-4925-966c-c0c231daa041", "level": "info", "msg": "Found 124 models, 492 tests, 0 snapshots, 0 analyses, 1189 macros, 2 operations, 6 seed files, 338 sources, 1 exposure, 0 metrics, 0 groups", "name": "FoundStats", "pid": 2968, "thread": "MainThread", "ts": "2024-03-08T12:40:51.910687Z"}}
{"data": {"msg": ""}, "info": {"category": "", "code": "Z017", "extra": {}, "invocation_id": "43f312f3-af1c-4925-966c-c0c231daa041", "level": "info", "msg": "", "name": "Formatting", "pid": 2968, "thread": "MainThread", "ts": "2024-03-08T12:40:51.930145Z"}}
{"data": {"node_count": 624, "num_threads": 2, "target_name": "dev"}, "info": {"category": "", "code": "Q027", "extra": {}, "invocation_id": "43f312f3-af1c-4925-966c-c0c231daa041", "level": "info", "msg": "Concurrency: 2 threads (target='dev')", "name": "ConcurrencyLine", "pid": 2968, "thread": "MainThread", "ts": "2024-03-08T12:40:53.397240Z"}}
{"data": {"msg": ""}, "info": {"category": "", "code": "Z017", "extra": {}, "invocation_id": "43f312f3-af1c-4925-966c-c0c231daa041", "level": "info", "msg": "", "name": "Formatting", "pid": 2968, "thread": "MainThread", "ts": "2024-03-08T12:40:53.398976Z"}}

Specifically, the line Extension executing `dbt compile`... is not in JSON format.

Code

No response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@menzenski