DIY: Equipment Data Logging & Analysis (Open Source)

I wanted an easy way to automatically, in real-time, measure, log and analyze equipment process metrics in order to learn from and improve efficiencies. Built on existing ‘off-the-shelf’ tools and a little custom software, I’ve put together a quick way for any lab or researcher to quickly digitize their existing lab equipment for active monitoring and analysis. Where my previous threads talked about specifics of hardware or software, I’ve put together this thread in hopes of explaining and providing the entire stack of software and hardware tools for free to others in the industry that can benefit from them. (All software and hardware documents can be found in the github link below)

Since I was starting from my personal interest to improve my short path distillation skills, the following example is monitoring a short path distillation system but can be applied to any set of devices, sensors, or metric. I’ve successfully used this system to monitor my short path many times as well as a custom made freeze dryer, co2 extractor, even my rosin press; anything we can apply a sensor to. In the case of the spd, I was able to get a stronger sense of what was happening to my head temps so that I could learn to cut better fractions. Instead of single data points, I was looking at data trends over time and could apply that to my operation.

**Please note that the following is a high-level technical guide to quickly get someone started with monitoring equipment metrics. This is in no way a fully secured, commercial grade, software solution but if that is something you seek, ( :electric_plug:) I am currently available for work or consulting and looking for interesting projects if anyone is in need of custom software/firmware, circuit/hardware design, 3d design, or other computer engineering concern. Happy to provide more background and references for my skillsets.

With that out of the way, lets talk about the software. The software stack utilizes 4 applications: Grafana, InfluxDB, Mosquito, and a custom node application named mqtt2influxdb. The overall idea is that your hardware will be sending messages over wifi to the MQTT broker, Mosquitto. The custom node application then takes the information from the MQTT broker and stores it in a database, InfluxDB. From here, we are able to view the time series data from a web browser logged into the Grafana interface. To get started, follow these steps:

  1. Install Docker and docker-compose
  2. Download git repo: https://github.com/dp621/connectedlab
  3. Docker-compose up in the folder
  4. Open browser to http://localhost:3000
  5. Login with admin/password

Congratulations! You now have a working software stack. Your new stack has been preconfigured with a database connection to influxdb and a fully working distillation dashboard that looks like this:

Don’t worry that there is no data available; now is when we can connect hardware devices to our system or run a simulated test using the provided tools. By running the provided node test application, spd_test.js, data is automatically sent to the MQTT broker as if a hardware device came online. Your distillation dashboard should now look like this:

For those that have been following this adventure, you have probably already seen my custom built vacuum monitor that reads mks 901p sensors. Since my last post here, I’ve made one final change to the device giving it a built in lcd display so that it can be used with or without the ConnectedLab software stack:


Complete schematics, circuit designs, firmware code, and case designs can also be found for immediate use in the above linked github repo under the /devices folder. I managed to order way too many circuits and parts so if you are in need of parts, a kit or fully built unit instead of sourcing the parts on your own, feel free to reach out in DM.

Any questions on the software stack reach out below!

21 Likes

Dude!!! I have always wanted to play with Grafana in a lab setting. I’m glad you took the initiative to integrate your data collection into software to log and work with it. This is SO badass man!

4 Likes

I like it, well done!

We’re using Grafana/MariaDB/Mosquitto here.

It’s usually sensor to PLCs to mqtt instead of sensor direct to mqtt, but it’s functionally the same thing.

4 Likes

Badass

1 Like

Are there PLC modules that are plug and play minus some setup? Definitely curious as to what others are already running.

Most of my hardware is built off either esp8266 or esp32 for a simple controller with wifi/ble and digital/analog io.

Pretty much everything meets that definition depending on how you mean plug and play and some setup.

They’re as simple or as complicated as you want them to be, but our systems are generally not exactly plug and play, until they’re fully built and integrated.

The one I’m working on presently has 60 I/O, with more to come. Though it’s reasonably simple from a controls perspective.

I think the simplest one we’re presently building is in the 25 I/O range, but the next iteration will probably come close to doubling that.

I did bang together a simple DAQ program that output to an HMI in a couple of hours, which is likely the closest approximation to what you’re doing.

2 Likes

We’ve been looking for software suites compatible with our new extraction flowmeter line, BHOmeters… there are some larger SaaS companies we’re having conversations with, but their subscription price might price out a lot of smaller operations that could still benefit from the data points our units provide… check out www.aBHOutit.com and dm me if you wanna have a brainstorm session.

Regardless. Really impressed by your work and effort. And especially your willingness to share.

3 Likes

Thanks and sent you a dm, always interested in a good technical convo.

Amazing work!

1 Like

I put together a spectrometer for use in the labs and in my grows/greenhouse. It was super short money, and could easily be worked into your stack. it could easily replace an aerometrix for half the cost and way better functionality and accuracy.

I would be willing to donate a s decent grip of money for a Individual that would open source a tool like this. I feel like a majority of the community may. In a world of green tax and pay to play … I would love to see more OS work.

6 Likes

Oooooo

1 Like

Does anybody know of an O2 sensor that could be threaded into a 1/4in npt port that would be ex proof? I am looking to add one to my reactor to ensure our sparging process is adequate.

1 Like

Could you use an automotive 02 sensor? Just spitballing

Unfortunately not. I believe those work using Titanium and they have to get HOT for it to read correctly.

Check out intrinsically safe store - senko makes one. It’s not NPT but you could likely make it work. Haven’t played with them myself - they don’t have the correct certs so we use indirect (overkill) methods for O2 reduction.

3 Likes