Skip to content

neptune-ai/kedro-neptune

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

58 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Neptune Kedro Integration

Kedro plugin for experiment tracking and metadata management. It lets you browse, filter and sort runs in a nice UI, visualize node/pipeline metadata, and compare pipelines.

What will you get with this integration?

  • browse, filter, and sort your model training runs
  • compare nodes and pipelines on metrics, visual node outputs, and more
  • display all pipeline metadata including learning curves for metrics, plots, and images, rich media like video and audio or interactive visualizations from Plotly, Altair, or Bokeh
  • and do whatever else you would expect from a modern ML metadata store

image Kedro pipeline metadata in custom dashboard in the Neptune UI

Note: Kedro-Neptune plugin supports distributed pipeline execution and works in Kedro setups that use orchestrators like Airflow or Kubeflow.

Resources

Example

# On the command line:
pip install neptune-client kedro kedro-neptune
kedro new --starter=pandas-iris

# In your Kedro project directory:
kedro neptune init
# In a pipeline node, in nodes.py:
import neptune.new as neptune

# Add a Neptune run handler to the report_accuracy() function
# and log metrics to neptune_run
def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame, 
                    neptune_run: neptune.run.Handler) -> None:
    target = np.argmax(test_y.to_numpy(), axis=1)
    accuracy = np.sum(predictions == target) / target.shape[0]
    
    neptune_run["nodes/report/accuracy"] = accuracy * 100

# Add the Neptune run handler to the Kedro pipeline
node(
    report_accuracy,
    ["example_predictions", "example_test_y", "neptune_run"],
    None,
    name="report")
# On the command line, run the Kedro pipeline
kedro run

Support

If you got stuck or simply want to talk to us, here are your options:

  • Check our FAQ page
  • You can submit bug reports, feature requests, or contributions directly to the repository.
  • Chat! When in the Neptune application click on the blue message icon in the bottom-right corner and send a message. A real person will talk to you ASAP (typically very ASAP),
  • You can just shoot us an email at [email protected]