Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mongo updates #1737

Merged
merged 2 commits into from
Feb 23, 2021
Merged

Mongo updates #1737

merged 2 commits into from
Feb 23, 2021

Conversation

varnerac
Copy link
Contributor

@varnerac varnerac commented Feb 22, 2021 •
edited
Loading

Proposed Changes

This PR does a few things.

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

  • Bugfix (non-breaking change which fixes issue VerneMQ Diversity Mongo Authentication Problem #1678)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation (correction or otherwise)
  • Cosmetics (whitespace, styles...)
  • DevOps (Build scripts, pipelines...)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating
the PR. If you're unsure about any of them, don't hesitate to ask on the
mailing list. We're here to help! This is simply a reminder of what we are
going to look for before merging your code.

  • I have read the CODE_OF_CONDUCT.md document
  • All tests pass locally with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if needed)
  • Any dependent changes have been merged and published in related repositories
  • I have updated changelog (At the bottom of the release version)
  • I have squashed all my commits into one before merging

Further Comments

This bumps rebar3 like #1731 did. I am happy to back that part out if needed.

The CI profile is a temporary workaround to get us some more testing in CI. Eventually we need all the things tested in Actions. Then, we can ditch the CI profile in for the all tests profile. However, I think this is definitely better than the status quo.

I'll add the DNS SRV documentation if this gets approved. The DNS SRV stuff and the latest MongoDB driver will help if we tackle #1471.

This also probably fixes #1117. I tested my MongoDB changes DNS SRV explicitly against Atlas Cloud databases and it worked in driver tests.

There's no good way to test the SRV functionality in CI because it depends on DNS records.

The diversity suite completely passes locally with the Docker Compose file.

varnerac@Drews-MacBook-Pro vernemq % ./rebar3 as all_tests ct --suite apps/vmq_diversity/test/vmq_diversity_provider_SUITE.erl
===> Verifying dependencies...
===> Running neotoma...
===> Analyzing applications...
===> Compiling vmq_acl
===> Compiling vmq_diversity
===> Compiling vmq_generic_msg_store
===> Compiling vmq_mqtt5_demo_plugin
===> Compiling vmq_passwd
===> Compiling vmq_plugin
===> Compiling vmq_plumtree
===> Compiling vmq_pulse
===> Compiling vmq_swc
===> Compiling vmq_webhooks
===> Compiling vmq_ql
===> Compiling vmq_server
===> Compiling vmq_commons
===> Compiling vmq_bridge
cc   -L/usr/local/opt/openssl/lib -I/usr/local/opt/openssl/include vmq_passwd.c -lcrypto -o ../priv/vmq_passwd
===> Running Common Test suites...
%%% vmq_diversity_provider_SUITE: ............
All 12 tests passed.

@ioolkos
Copy link
Contributor

ioolkos commented Feb 22, 2021

very welcome improvements on MongoDB :)
review upcoming tomorrow, thanks!

@varnerac varnerac force-pushed the mongo_updates branch 2 times, most recently from 2ac2add to 6dc6e40 Compare February 23, 2021 16:29
@varnerac
Copy link
Contributor Author

That's as much as I can fix in the the CI runs now. If you re-run, it should pass

@varnerac
Bump MongoDB driver to upstream with SRV
dde6c11 
Bumps the MongoDB driver to latest upstream. Adds DNS SRV support to
MongoDB in vmq_diversity. Add authentication to MongoDB test. Adds
Docker Compose file for local testing. New CI profile to get tests with
external services running in GitHub Actions. Bumps `rebar3` binary to
latest.
ioolkos
ioolkos reviewed Feb 23, 2021
View reviewed changes
apps/vmq_diversity/src/vmq_diversity_sup.erl
SslOpts = proplists:get_value(ssl_opts, WorkerArgs0),
WorkerArgs1 = proplists:delete(ssl, WorkerArgs0),
WorkerArgs2 = proplists:delete(ssl_opts, WorkerArgs1),
mc_worker_api:connect([{ssl,Ssl}, {ssl_opts, SslOpts}|WorkerArgs2])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does mc_worker_api:connect/1 change anything in behaviour? vs. mc_worker:start_link/1?
(fine with me, I'm just wondering what the difference is as the params are the same.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. It's a thin wrapper around mc:worker:start_link/1.

If you pass login and password in the proplist, it authenticates the connection.

https://github.com/comtihon/mongodb-erlang/blob/master/src/api/mc_worker_api.erl#L64-L68

I am not sure how authentication was working without this.

ioolkos
ioolkos approved these changes Feb 23, 2021
View reviewed changes
Copy link
Contributor

@ioolkos ioolkos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jep, good for me including the rebar3 bump (worst case I can always go change that).
pkg_hash_ext is new in rebar.lock
Have not functionally tested the MongoDB stuff but looks good to merge.

@varnerac varnerac mentioned this pull request Feb 23, 2021
Closed
13 tasks
@ioolkos ioolkos merged commit de2e160 into vernemq:master Feb 23, 2021
sthagen added a commit to sthagen/vernemq-vernemq that referenced this pull request Feb 24, 2021
@sthagen
Merge pull request #33 from vernemq/master
e1e3105 
Bump MongoDB driver to upstream with SRV (vernemq#1737)
ioolkos added a commit that referenced this pull request Apr 10, 2021
@ioolkos @varnerac
@elduds @rubengees @steve999lee
Feature rocksdb messagestore (#1774)
cb676f3 
* Update THANKS

* Changing default: Don't make bridge direct_plugin_exports (subscribe,… (#1629)

* Changing default: Don't make bridge direct_plugin_exports (subscribe, publish etc) wait for cluster readiness

* Routing through QoS to internal PublishFun

* Update changelog.md

* Fix handling of WebSocket protocols (#1635)

Correctly handles mutliple values in the `sec_websocket_protocol` HTTP
header.

* Update rebar.lock

* Update rebar.config

* Update changelog.md

* Improve Proxy protocol error logging (#1637)

* Adds usage information for vmq-admin retain command (#1) (#1639)

* vmq-admin usage() includes retain command

* Update changelog.md

* Update vmq_diversity.schema

* Update changelog.md

* Update changelog.md

* Fix size for database pools not being honored (#1657)

* Ensure_that_MQTT_v5_user_properties_are_stored_to_disk/DB_#1560 (#1655)

* Ensure_that_MQTT_v5_user_properties_are_stored_to_disk/DB_with_a_message_#1560

* relax p_msg_val_pre() type to allow properties to be persisted with payload

Co-authored-by: ioolkos <[email protected]>

* Fix RPC query

* Add "m" shortcut for "more"

* update certifi to 2.5.3 (#1685)

* Allow configuration of buffer sizes for vmq_cluster_com, and autotune 'buffer' for vmq_cluster_node

* no need for masking

* Make Watermarks configurable for vmq_cluster_com listener

* Update Hackney to 1.17.0 [dependency]

* Update changelog.md

* Webhooks HTTPS Enhancements

Adds HTTPS-specific options to the webhooks module and tests. Allows
users to specify a custom CA and verification options. Allows hooks
to authenticate to endpoints with client certificates.

Adds PEM validation to the vmq_server and vmq_webhooks schemas.

* Updated clique dependency to fix trailing equal sign (Issue #1673)

* Add missing server cert for webhooks test suite

* Fix verify function

Corrects the verify function to take the endpoint hostname. Sets
reuse_sessions to false when verifying the peer certificate.

* Add GitHub Actions

* Update index.html

Remove Slack

* Webhooks HTTPS improvements (#1711)

* Webhooks HTTPS improvements

This adds CRL checking, which is enabled by default.

It adds a number of tests to HTTPS Webhooks functionality, including:

* fail if CRLs are required but not available
* fail if the endpoint certificate is expired
* fail if the endpoint certificate Common Name does not match hostname

* Update index.html

Remove Slack

* Bump to trigger CI

Co-authored-by: ioolkos <[email protected]>

* Add macOS to CI (#1715)

* Create ADOPTERS.md

* Bump MongoDB driver to upstream with SRV (#1737)

Bumps the MongoDB driver to latest upstream. Adds DNS SRV support to
MongoDB in vmq_diversity. Add authentication to MongoDB test. Adds
Docker Compose file for local testing. New CI profile to get tests with
external services running in GitHub Actions. Bumps `rebar3` binary to
latest.

* Revert rebar3 (#1740)

Reverts rebar3 to earlier version to fix broken release. Adds a smoke
test in GitHub actions to detect broken releases in CI.

* Update internal Luerl interface (luerl:do/1 is no more), update Luerl dependency (#1745)

* Update changelog.md

* upgrade to Cowboy 2.8.0 (#1747)

* upgrade to Cowboy 2.8.0

* Add upgrade note to changelog.

* Ignores the top-level data dir (#1749)

* Support the MongoDB auth_source connect option (#1750)

* Safe binary to term (#1752)

* Validate UTF8 strings in topics and use safe binary_to_term.

* Update changelog

* update hackney to 1.17.4 (#1756)

* fix ws options in proxy_protocol case (#1739)

* fix ws options in proxy_protocol case

* get peer info from proxy info

* basic connect test for ws proxy protocol

* add test for proxy protocol local command

* Update changelog.md

* Create FUNDING.yml

* Update deps (#1772)

* update hackney to 1.17.4

* pin rebar3_cuttlefish to commit tag

* dialyzer

Co-authored-by: Andrew Varner <[email protected]>
Co-authored-by: Luke Dudney <32501671 [email protected]>
Co-authored-by: Ruben Gees <[email protected]>
Co-authored-by: Steve Lee <[email protected]>
Co-authored-by: Drew Varner <[email protected]>
Co-authored-by: wojciech.luczkow <[email protected]>
codeadict pushed a commit to codeadict/vernemq that referenced this pull request Feb 4, 2022
@ioolkos @varnerac
@elduds @rubengees @steve999lee @codeadict
Feature rocksdb messagestore (vernemq#1774)
a668f85 
* Update THANKS

* Changing default: Don't make bridge direct_plugin_exports (subscribe,… (vernemq#1629)

* Changing default: Don't make bridge direct_plugin_exports (subscribe, publish etc) wait for cluster readiness

* Routing through QoS to internal PublishFun

* Update changelog.md

* Fix handling of WebSocket protocols (vernemq#1635)

Correctly handles mutliple values in the `sec_websocket_protocol` HTTP
header.

* Update rebar.lock

* Update rebar.config

* Update changelog.md

* Improve Proxy protocol error logging (vernemq#1637)

* Adds usage information for vmq-admin retain command (#1) (vernemq#1639)

* vmq-admin usage() includes retain command

* Update changelog.md

* Update vmq_diversity.schema

* Update changelog.md

* Update changelog.md

* Fix size for database pools not being honored (vernemq#1657)

* Ensure_that_MQTT_v5_user_properties_are_stored_to_disk/DB_#1560 (vernemq#1655)

* Ensure_that_MQTT_v5_user_properties_are_stored_to_disk/DB_with_a_message_#1560

* relax p_msg_val_pre() type to allow properties to be persisted with payload

Co-authored-by: ioolkos <[email protected]>

* Fix RPC query

* Add "m" shortcut for "more"

* update certifi to 2.5.3 (vernemq#1685)

* Allow configuration of buffer sizes for vmq_cluster_com, and autotune 'buffer' for vmq_cluster_node

* no need for masking

* Make Watermarks configurable for vmq_cluster_com listener

* Update Hackney to 1.17.0 [dependency]

* Update changelog.md

* Webhooks HTTPS Enhancements

Adds HTTPS-specific options to the webhooks module and tests. Allows
users to specify a custom CA and verification options. Allows hooks
to authenticate to endpoints with client certificates.

Adds PEM validation to the vmq_server and vmq_webhooks schemas.

* Updated clique dependency to fix trailing equal sign (Issue vernemq#1673)

* Add missing server cert for webhooks test suite

* Fix verify function

Corrects the verify function to take the endpoint hostname. Sets
reuse_sessions to false when verifying the peer certificate.

* Add GitHub Actions

* Update index.html

Remove Slack

* Webhooks HTTPS improvements (vernemq#1711)

* Webhooks HTTPS improvements

This adds CRL checking, which is enabled by default.

It adds a number of tests to HTTPS Webhooks functionality, including:

* fail if CRLs are required but not available
* fail if the endpoint certificate is expired
* fail if the endpoint certificate Common Name does not match hostname

* Update index.html

Remove Slack

* Bump to trigger CI

Co-authored-by: ioolkos <[email protected]>

* Add macOS to CI (vernemq#1715)

* Create ADOPTERS.md

* Bump MongoDB driver to upstream with SRV (vernemq#1737)

Bumps the MongoDB driver to latest upstream. Adds DNS SRV support to
MongoDB in vmq_diversity. Add authentication to MongoDB test. Adds
Docker Compose file for local testing. New CI profile to get tests with
external services running in GitHub Actions. Bumps `rebar3` binary to
latest.

* Revert rebar3 (vernemq#1740)

Reverts rebar3 to earlier version to fix broken release. Adds a smoke
test in GitHub actions to detect broken releases in CI.

* Update internal Luerl interface (luerl:do/1 is no more), update Luerl dependency (vernemq#1745)

* Update changelog.md

* upgrade to Cowboy 2.8.0 (vernemq#1747)

* upgrade to Cowboy 2.8.0

* Add upgrade note to changelog.

* Ignores the top-level data dir (vernemq#1749)

* Support the MongoDB auth_source connect option (vernemq#1750)

* Safe binary to term (vernemq#1752)

* Validate UTF8 strings in topics and use safe binary_to_term.

* Update changelog

* update hackney to 1.17.4 (vernemq#1756)

* fix ws options in proxy_protocol case (vernemq#1739)

* fix ws options in proxy_protocol case

* get peer info from proxy info

* basic connect test for ws proxy protocol

* add test for proxy protocol local command

* Update changelog.md

* Create FUNDING.yml

* Update deps (vernemq#1772)

* update hackney to 1.17.4

* pin rebar3_cuttlefish to commit tag

* dialyzer

Co-authored-by: Andrew Varner <[email protected]>
Co-authored-by: Luke Dudney <32501671 [email protected]>
Co-authored-by: Ruben Gees <[email protected]>
Co-authored-by: Steve Lee <[email protected]>
Co-authored-by: Drew Varner <[email protected]>
Co-authored-by: wojciech.luczkow <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ioolkos ioolkos ioolkos approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

VerneMQ Diversity Mongo Authentication Problem crash when trying to connect with mongo atlas cluster
2 participants
@varnerac @ioolkos