Groq is one of the fastest inference platforms in AI. Command Line Interfaces (CLI) are very fast when all you have work with is text. Terminal user interfaces are boring by default. However, terminal coloring can be used to make CLI look more interesting. This project combined these factors to create a very good user experience.
Python was used as the language for implementing the app. The Groq SDK was used for communicating with the Groq inference platform. Rich was used for coloring the headers for the AI and the user. Symbols were used to represent the chat bot and the user. The app uses emojis to for symbols by default. The app also gives the user the option to use a nerd font for the symbols.
The following environment variables were used:
GROQ_API_KEY
: the API key used to access Groq services.MODEL
: the AI model to use for the inference.Data classes and type hints were used to create entities. The following sections give an overview of the entities that were involved.
The Message
data class was used to create messages used in the chat.
@dataclass(frozen = True) class Message: role: str # 'system' | 'assistant' | 'user' content: str
This data class was used for terminal coloring using Rich.
@dataclass(frozen = True) class TextStyles: foreground: str background: str @property def foreground_and_background(self) -> str: return f'bold {self.foreground} on {self.background}'
The Header
data class was used for displaying the header of the AI and user messages.
@dataclass(frozen = True) class Header: using_nerd_font: bool role: str text: str def __repr__(self): if self.using_nerd_font: return self.format_using_nerd_font() else: return self.format_without_nerd_font() def format_using_nerd_font(self) -> str: icon = self.get_icon(nerd_font_icons) heading = f'{icon} {self.text}' styles = text_styles[self.role] open = Header.set_text_styles(styles.background, '') close = Header.set_text_styles(styles.background, '') middle = Header.set_text_styles(styles.foreground_and_background, heading) return f'{open}{middle}{close}' def format_without_nerd_font(self) -> str: styles = text_styles[self.role].foreground_and_background return Header.set_text_styles(styles, f' @{self.text} ') def get_icon(self, icons: Icons) -> str: return icons.user if self.role == 'user' else icons.assistant @classmethod def set_text_styles(cls, styles: str, text: str) -> str: return f'[{styles}]{text}[/]'
The following code was used to run the app:
def main(): handle_missing_env_var('GROQ_API_KEY', GROQ_API_KEY) handle_missing_env_var('MODEL', MODEL) print("Enter 'exit' to quit chat.\n") console.print(headers['assistant']) print('How may I assist you?') print() while True: user_message = get_user_message() if user_message.content == 'exit': break messages.append(asdict(user_message)) print() ai_message = get_ai_message() messages.append(asdict(ai_message)) print('\n')
The function handle_missing_env_var
checks if an environment variable is set. If that variable is not set, it
informs the user of that, and closes the app. The message sent by the user is gotten using get_user_message
. The function handles prompting the user and getting the response. If the user enters
exit
, the app is closed. AI messages were gotten using get_ai_message
. The function gets the
message using streaming and displays whatever content has been accumulated.
All the messages from the AI and the user are stored in a list for use in further prompts.
The app had the following strengths:
The app can be used for simple and fast inference. If you want quick answers to questions, instead of opening your browser, which is much slower, you could use your terminal instead.
For generative AI developers, this app can be used for learning how to create AI chat apps with streaming capabilities.
More error handling could be added to reduce surprises. The area that lacks error handling the most is the usage of the Groq SDK. Inference errors could be caught and investigated. Upon investigation, the user could be informed on what went wrong. Possible errors include network errors, exceeding the token window, and unknown errors.
Chat AI usually use markdown for the responses. The markdown could be formatted and highlighted. This would result in better looking messages which would provide a good user experience.
I had a lot of fun creating this project. The experience was much better than I expected. The Groq inference platform really delivers on its promises. AI chat apps have gotten much faster these days. But, at the time I started this project, they were very slow. Seeing how fast the app could display messages, I was amazed. The app is open sourced on GitHub at mishieck/groq-python-chat.
There are no datasets linked
There are no models linked
There are no models linked
There are no datasets linked