I've put out a PR (#779) that brings up each node type with proper communication throughout. with a singled docker compose up command. There are some manual steps still to get everything running, copied below.

I'd love to discuss making it more seamless to bring the whole stack locally. Improvements to the instructions below would be:

1. removing the manual step (2) below. I think fixing the underlying issue makes the most sense since topic test-topic-in is not supported externally anyway. (see updates to dockerfile to run astra and dependencies locally #779 (comment))
2. Automatically performing step (3) locally via script or some other way.

Instructions for local development.

Putting these in the PR description for now, but potentially this should be added to the Onboarding docs or a new local development doc.

1. in terminal, build and run compose to bring up dependencies and kaldb nodes

docker build -t slackhq/astra .

docker compose up

2. in kafka container terminal, create input topic (preprocessor crashes if it does not exist before configuring manager in next step)

kafka-topics.sh --create --topic test-topic-in --bootstrap-server localhost:9092

3. run 2 curl commands to configure 1 partition

curl -XPOST -H 'content-type: application/json; charset=utf-8; protocol=gRPC' http://localhost:8083'/slack.proto.astra.ManagerApiService/CreateDatasetMetadata' -d '{
  "name": "test",
  "owner": "[email protected]",
  "serviceNamePattern": "_all"
}'

curl -XPOST -H 'content-type: application/json; charset=utf-8; protocol=gRPC' http://localhost:8083'/slack.proto.astra.ManagerApiService/UpdatePartitionAssignment' -d '{
  "name": "test",
  "throughputBytes": "1000000",
  "partitionIds": ["0"]
}'

4. Add some logs via bulk ingest

curl --location 'http://localhost:8086/_bulk' \
--header 'Content-type: application/x-ndjson' \
--data '{ "index" : { "_index" : "test", "_id" : "100" } }
{ "@timestamp": "2024-03-07T12:00:00.000Z", "level": "INFO", "message": "This is a log message", "service-name": "test" }
'

5. Example curl to read data

Note: This is similar to ES _msearch but size, lte, and gte are all required or else a 500 error occurs. In follow-up work, I will clean up those endpoints to return a 400 (bad request) with detail about the missing required argument. PR around follow-up: #780

curl --location 'http://localhost:8081/_msearch' \
--header 'Content-Type: application/x-ndjson' \
--data-binary $'{ "index": "test"}\n{"query": {"range": {"@timestamp": {"gte": 1625156649889,"lte": 2708540790265}}}, "size": 500}\n' | jq .