ChainForge is an open-source visual programming Interface for LLM flows and more…
I’m currently the Chief Evangelist @ HumanFirst. I explore & write about all things at the intersection of AI & language. Including NLU design, evaluation & optimisation. Data-centric prompt tuning & LLM observability, evaluation & fine-tuning.
For tips on installation, go to the end of this article.
The objective of ChainForge is the comparing and evaluation of prompts and model responses.
Users will most probably construct a quick flow to examine prompt structure and compare the response of different models.
ChainForge is premised on the following three tasks:
With ChainForge you can setup prompt templates, chain multiple prompts together, as seen below. Multiple methods can be used for prompt template input:
- CSV Nodes,
- Python output, etc.
Consider the example below where a CSV node populates a prompt template, chained to a prompt with further instructions for the LLM.
Evaluation nodes are Python script -based components, which test LLM responses classically for expected behaviour.
Considering the image below, where GPT3.5 and GPT4 are tested for prompt injection vulnerability.
Two nodes are used, both holding a set of instructions.
Notice the prompt node, the simple Python evaluation script, and the output.
From this example it’s clear that GPT3.5’s susceptibility to prompt injection is higher.
The default visualisation features are quite good, as can be seen below. The level of visualisation is highly dependant on the data processing performed in the Python script.
Considering the image below, if you return a dictionary with more than one key, metrics will be plotted in a parallel coordinates plot.
LangFlow and Flowise are both developed as fully fledge chaining applications with Agent capability; both leveraging LangChain.
As mentioned earlier, ChainForge was built for specific purposes and will also work well as a playground for:
- Model Testing,
- Model Comparison,
- Prompt Engineering & Templating
- Data Visualisation
Additional features which would help are:
The addition of tabs will be a great help, allowing users to have multiple flows open and use tabs to switch between flows.
Exposing flows via an API will also be helpful, currently flows can be exported and imported; hence being shared as ASCII files. But API functionality will help.
Currently ChainForge only have seven components, listed below. With Export & Import options. An improvement will be to have a pallet or pane on the left, with the components listed and a description on their functionality.
Only two commands are required to install and run ChainForge:
pip install chainforge &
When installing ChainForge via the MacOS Terminal, you will most probably run into the following error:
note: This error originates from a subprocess, and is likely not a problem with pip.
× Encountered error while trying to install package.
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
To solve this, you will need to run
xcode-select --install . Once installation is successful, and the
serve command is run, the following is shown in the terminal:
Serving SocketIO server on port 8001...
Serving Flask server on port 8000...
* Serving Flask app 'chainforge.flask_app'
* Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
* Running on http://localhost:8000
As seen below, the only configuration you will have to perform is adding the LLM API Key detail for your model.
I’m currently the Chief Evangelist @ HumanFirst. I explore and write about all things at the intersection of AI and language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces and more.
NLU design tooling
“Conversation Designer, Retail, 10k+ employees The tool that turned conversation designers, into NLU designers” ★★★★★…
Get an email whenever Cobus Greyling publishes.
Get an email whenever Cobus Greyling publishes. By signing up, you will create a Medium account if you don’t already…
At the intersection of AI & Language | NLP/NLU/LLM, Chat/Voicebots, CCAI Chief Evangelist @ HumanFirst. I explore and…
GitHub - ianarawjo/ChainForge: An open-source visual programming environment for battle-testing…
ChainForge is a data flow prompt engineering environment for analyzing and evaluating LLM responses. It is geared…
ChainForge/examples at main · ianarawjo/ChainForge
You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or…