Your Questions Answered
Our Lobster_live session on external monitoring with Grafana and New Relic, hosted by Jannick Fertl on August 28, brought in more questions than we had time to answer live. Here are the answers, grouped by topic.
The metrics in the Lobster Platform (Cloud) Service portal cover your AWS installation and differ slightly from the metrics you see in external monitoring. Most of them overlap.
The „monitoring“ metrics themselves give you live data from the system. If your external monitoring tool collects that data over time, you get historical data through the tool. Depending on how long your Lobster Data Platform system retains statistical data, you can also view some history directly on the system.
Not at the moment. Detailed documentation of the provided KPIs and metrics does not exist yet.
ldp.errors.total tracks total job errors on the system. There is currently no separate metric for errors that remain unresolved.
Not into the automated metrics connection itself. You can, however, send custom process data to your external monitoring tool through the endpoints most monitoring tools provide for this purpose. Both New Relic and Grafana support custom data through their own endpoints.
The built-in monitoring doesn’t currently break down stats by individual profile or workflow. You can work around this by editing a profile to send an HTTP POST at the relevant point, so the external monitoring tool captures that specific event.
Not currently planned.
Profile logs can be exported through a custom .jar file, which you import into the Lobster file server and configure to specify which log files or database entries get sent to external monitoring.
The Monitoring menu itself doesn’t support this. You can build a profile that triggers on error and sends the message directly to the Slack API.
Two options: monitor the error log in your external tool via the custom .jar file mentioned above, or set up an Error Handler Profile that triggers on any error and creates the alert.
Yes, you can connect any number of LDP instances to one monitoring tool. Use a distinct „Service name“ or „System environment“ in the Monitoring configuration for each instance so you can tell them apart in New Relic or Grafana.
Yes. You can set up as many monitoring configurations as you like, and they all run and send data in parallel.
It depends on your subscription tier and the tool you choose. Some offer free tiers or trials, but running one of these tools generally adds cost.
